defnoteatin commited on
Commit
6f6bbdd
·
verified ·
1 Parent(s): 2602516

Update README.md and pb27_vllm_plugin (dense global agg, non-overlapping RoPE, resilient vLLM) for defnoteatin/PseudoBrain-Qwen3-Coder-Next-BF16

Browse files
README.md CHANGED
@@ -38,8 +38,10 @@ In standard Transformer serving, **model weights are paid once, while KV cache m
38
 
39
  **`PseudoBrain-Qwen3-Coder-Next-BF16`** (`PB27Qwen3NextForCausalLM`) solves this via the **PseudoBrain architecture**:
40
  1. **Hybrid Windowed-Attention + Parallel Gated DeltaNet (`GDN`):** 48 layers: 36 native Gated DeltaNet + 12 full-attention layers (`[3, 7, 11, ..., 47]`) converted into 16K SWA + Parallel Gated DeltaNet + Quest Exact Memory. Recurrent GDN state is **$O(1)$ fixed size** regardless of whether context is `4K` or `1,000,000` tokens.
41
- 2. **Three-Tier NVMe/Page-Cache Quest Exact Memory (`piece_attention`):** On the sparse anchor layers, queries attend over `[4 StreamingLLM Sinks + Top-k Quest Exact-Memory Pages + 16,384 Sliding Window]` via FlashAttention-2 LSE-merge (`pb27_vllm/memory.py`) with **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`)** and **Priority-Aware RoPE Sink Repositioning (`_reposition_rope_sinks`)**.
42
- 3. **True $O(1)$ Constant-Speed Decode up to `1,000,000` Tokens:** Decouples the scaled prefill retrieval budget from the constant decode budget (`PB27_MEMORY_DECODE_STRIDE=64`, `PB27_MEMORY_DECODE_BLOCKS=64`, `PB27_MEMORY_DECODE_RERANK=0`), cutting **1M-token KV cache memory by `93.5%`** (`~48.0 GB` → **`~3.1 GB`**).
 
 
43
 
44
  ### Hardware & Precision Editions for `PseudoBrain-Qwen3-Coder-Next`
45
 
 
38
 
39
  **`PseudoBrain-Qwen3-Coder-Next-BF16`** (`PB27Qwen3NextForCausalLM`) solves this via the **PseudoBrain architecture**:
40
  1. **Hybrid Windowed-Attention + Parallel Gated DeltaNet (`GDN`):** 48 layers: 36 native Gated DeltaNet + 12 full-attention layers (`[3, 7, 11, ..., 47]`) converted into 16K SWA + Parallel Gated DeltaNet + Quest Exact Memory. Recurrent GDN state is **$O(1)$ fixed size** regardless of whether context is `4K` or `1,000,000` tokens.
41
+ 2. **Three-Tier NVMe/Page-Cache Quest Exact Memory (`piece_attention`):** On the sparse anchor layers, queries attend over `[4 StreamingLLM Sinks + Super-Block Landmarks + Top-k Quest Exact-Memory Pages + 16,384 Sliding Window]` via FlashAttention-2 LSE-merge (`pb27_vllm/memory.py`) with **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`)**.
42
+ 3. **Piecewise-Proportional, Strictly Non-Overlapping Block RoPE (`_reposition_rope_blocks`):** Enforces a mandatory `64`-token footprint per retrieved block plus proportional historical-gap scaling ($T(b_j) = P_{ ext{start}} + j \cdot B + ext{round}( rac{G_{ ext{cum},j}}{G_{ ext{tot}}} S_{ ext{slack}})$) — preserving exact `+1` local continuity across contiguous block runs, strictly preventing phase overlap between nearby non-adjacent blocks, and maintaining **99%+ proportional macro-timeline distance fidelity** at `1M` tokens without static YaRN degradation.
43
+ 4. **Dense Global Aggregation (`PB27_MEMORY_GLOBAL_AGG="auto"`):** Automatically detects high-participation-ratio global queries (`_detect_global_agg`) and activates **(a) Resident Super-Block Summary Landmarks (`smean_k`, `smean_v`, 1 summary token per `4,096` tokens in GPU HBM)**, **(b) Stratified Super-Block Timeline Coverage**, and **(c) Novelty-Rotating Decode Working-Set Sweeps (`visit_counts`)** so multi-step reasoning sweeps across hundreds of distinct blocks over `1M` tokens.
44
+ 5. **True $O(1)$ Constant-Speed Decode & Version-Resilient `vLLM` Plugin:** Decouples the scaled prefill retrieval budget from the constant decode budget (`PB27_MEMORY_DECODE_STRIDE=64`, `PB27_MEMORY_DECODE_BLOCKS=64`), cutting **1M-token KV cache memory by `93.5%`** (`~48.0 GB` → **`~3.1 GB`**), with dual `vLLM` V1/V2 runner support and a zero-hook `ForwardContext` fallback.
45
 
46
  ### Hardware & Precision Editions for `PseudoBrain-Qwen3-Coder-Next`
47
 
pb27_vllm_plugin/pb27_vllm/__init__.py CHANGED
@@ -26,54 +26,102 @@ UNIVERSAL_ARCHS = (
26
  )
27
 
28
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
  def register() -> None:
 
 
30
  from vllm import ModelRegistry
31
- from vllm.config import engram as vllm_engram
32
- from vllm.config import vllm as vllm_cfg
33
- from vllm.model_executor.models.config import (
34
- MODELS_CONFIG_MAP,
35
- Gemma4Config,
36
- Qwen3_5ForCausalLMConfig,
37
- Qwen3_5ForConditionalGenerationConfig,
38
- _strip_qwen4_exp_mrope,
39
- )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
 
41
  class PB27Gemma4Config(Gemma4Config):
42
  @staticmethod
43
  def verify_and_update_config(vllm_config) -> None:
44
  Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
45
- Gemma4Config.verify_and_update_config(vllm_config)
 
46
 
47
  class PB27Qwen4ExpForConditionalGenerationConfig(
48
  Qwen3_5ForConditionalGenerationConfig
49
  ):
50
- """Qwen4ExpForConditionalGenerationConfig (config.py:876-916) plus
51
- DFlash2 speculative decoding support."""
52
 
53
  @staticmethod
54
  def verify_and_update_config(vllm_config) -> None:
55
  Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
56
  text_config = vllm_config.model_config.hf_text_config
57
- if text_config.hc_count <= 1:
58
  raise ValueError("Qwen4Exp requires hc_count > 1")
59
  parallel_config = vllm_config.parallel_config
60
- uses_ple_or_qsa = bool(text_config.ple_layer_ids) or (
61
  getattr(text_config, "indexer_n_heads", None) is not None
62
  )
63
  if uses_ple_or_qsa and (
64
- parallel_config.enable_dbo or parallel_config.ubatch_size > 1
 
65
  ):
66
  raise NotImplementedError(
67
  "Qwen4Exp PLE/QSA does not support dual-batch overlap or microbatching"
68
  )
69
- if text_config.ple_layer_ids and parallel_config.pipeline_parallel_size > 1:
70
  raise NotImplementedError(
71
  "Qwen4Exp N-gram PLE embedding requires pipeline_parallel_size=1 "
72
  "because non-first pipeline ranks do not receive the raw input_ids "
73
  "it needs. Please run with PP=1."
74
  )
75
  multimodal_config = vllm_config.model_config.multimodal_config
76
- if multimodal_config is not None and multimodal_config.language_model_only:
77
  _strip_qwen4_exp_mrope(vllm_config.model_config)
78
  spec_config = vllm_config.speculative_config
79
  if spec_config is not None:
@@ -102,28 +150,28 @@ def register() -> None:
102
  )
103
  _strip_qwen4_exp_mrope(vllm_config.model_config)
104
 
 
105
  for arch in ARCHS:
106
- if arch not in ModelRegistry.get_supported_archs():
107
  ModelRegistry.register_model(arch, f"pb27_vllm.model:{arch}")
108
  for arch in LLAMA_ARCHS:
109
- if arch not in ModelRegistry.get_supported_archs():
110
  ModelRegistry.register_model(arch, f"pb27_vllm.llama_model:{arch}")
111
  for arch in QWEN4_EXP_ARCHS:
112
- if arch not in ModelRegistry.get_supported_archs():
113
  ModelRegistry.register_model(arch, f"pb27_vllm.qwen4_exp_model:{arch}")
114
  for arch in UNIVERSAL_ARCHS:
115
- if arch not in ModelRegistry.get_supported_archs():
116
  ModelRegistry.register_model(arch, f"pb27_vllm.universal_models:{arch}")
117
 
118
  # The verify-and-update hooks of the stock Qwen3.5 classes (mamba SSM cache
119
  # dtype from the HF config; the CausalLM variant also drops the M-RoPE
120
  # fields). PB-27 keeps model_type qwen3_5(_moe) / qwen3_5(_moe)_text so they
121
- # apply as is; the stock MoE classes map to the same hooks (config.py:1061-1062).
122
  for arch in ("PB27ForConditionalGeneration", "PB27MoeForConditionalGeneration"):
123
  MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForConditionalGenerationConfig)
124
  for arch in ("PB27ForCausalLM", "PB27MoeForCausalLM"):
125
  MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForCausalLMConfig)
126
- # The Llama / Qwen3Next / Qwen3Moe / Qwen3 paths only need the mamba SSM cache dtype hook (no M-RoPE fields to drop).
127
  for arch in (
128
  *LLAMA_ARCHS,
129
  "PB27Qwen3NextForCausalLM",
@@ -142,12 +190,17 @@ def register() -> None:
142
  PB27Qwen4ExpForCausalLMConfig,
143
  )
144
 
145
- for arch in QWEN4_EXP_ARCHS:
146
- vllm_engram._NGRAM_LAYER_FIELDS[arch] = "ple_layer_ids"
147
- vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES = (
148
- vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES | frozenset(QWEN4_EXP_ARCHS)
149
- )
150
- vllm_cfg.default_breakable_cudagraph_architectures.cache_clear()
 
 
 
 
 
151
 
152
  # Exact memory tier (PB27_MEMORY env): archive bookkeeping hooks on the GPU model runner.
153
  from . import memory
@@ -157,6 +210,4 @@ def register() -> None:
157
  try:
158
  memory.install_runner_hooks()
159
  except Exception as exc: # the API-server process has no runner; the engine/worker processes do
160
- import logging
161
-
162
- logging.getLogger(__name__).info("pb27 memory hooks not installed here: %s", exc)
 
26
  )
27
 
28
 
29
+ def _fallback_strip_mrope(model_config) -> None:
30
+ """Version-tolerant fallback if vLLM moves or renames `_strip_qwen4_exp_mrope`."""
31
+ hf_text = getattr(model_config, "hf_text_config", None)
32
+ for cfg in (getattr(model_config, "hf_config", None), hf_text):
33
+ if cfg is None:
34
+ continue
35
+ for container_name in ("rope_parameters", "rope_scaling"):
36
+ rope_cfg = getattr(cfg, container_name, None)
37
+ if isinstance(rope_cfg, dict):
38
+ rope_cfg.pop("mrope_section", None)
39
+ rope_cfg.pop("mrope_interleaved", None)
40
+
41
+
42
  def register() -> None:
43
+ import logging
44
+
45
  from vllm import ModelRegistry
46
+
47
+ log = logging.getLogger(__name__)
48
+
49
+ try:
50
+ from vllm.config import engram as vllm_engram
51
+ except Exception:
52
+ vllm_engram = None
53
+
54
+ try:
55
+ from vllm.config import vllm as vllm_cfg
56
+ except Exception:
57
+ vllm_cfg = None
58
+
59
+ try:
60
+ from vllm.model_executor.models.config import MODELS_CONFIG_MAP
61
+ except Exception:
62
+ MODELS_CONFIG_MAP = {}
63
+
64
+ try:
65
+ from vllm.model_executor.models.config import (
66
+ Qwen3_5ForCausalLMConfig,
67
+ Qwen3_5ForConditionalGenerationConfig,
68
+ )
69
+ except Exception:
70
+ class Qwen3_5ForConditionalGenerationConfig: # type: ignore[no-redef]
71
+ @staticmethod
72
+ def verify_and_update_config(vllm_config) -> None:
73
+ pass
74
+
75
+ class Qwen3_5ForCausalLMConfig(Qwen3_5ForConditionalGenerationConfig): # type: ignore[no-redef]
76
+ pass
77
+
78
+ try:
79
+ from vllm.model_executor.models.config import Gemma4Config
80
+ except Exception:
81
+ Gemma4Config = Qwen3_5ForConditionalGenerationConfig
82
+
83
+ try:
84
+ from vllm.model_executor.models.config import _strip_qwen4_exp_mrope
85
+ except Exception:
86
+ _strip_qwen4_exp_mrope = _fallback_strip_mrope
87
 
88
  class PB27Gemma4Config(Gemma4Config):
89
  @staticmethod
90
  def verify_and_update_config(vllm_config) -> None:
91
  Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
92
+ if Gemma4Config is not Qwen3_5ForConditionalGenerationConfig:
93
+ Gemma4Config.verify_and_update_config(vllm_config)
94
 
95
  class PB27Qwen4ExpForConditionalGenerationConfig(
96
  Qwen3_5ForConditionalGenerationConfig
97
  ):
98
+ """Qwen4ExpForConditionalGenerationConfig plus DFlash2 speculative decoding support."""
 
99
 
100
  @staticmethod
101
  def verify_and_update_config(vllm_config) -> None:
102
  Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
103
  text_config = vllm_config.model_config.hf_text_config
104
+ if getattr(text_config, "hc_count", 4) <= 1:
105
  raise ValueError("Qwen4Exp requires hc_count > 1")
106
  parallel_config = vllm_config.parallel_config
107
+ uses_ple_or_qsa = bool(getattr(text_config, "ple_layer_ids", None)) or (
108
  getattr(text_config, "indexer_n_heads", None) is not None
109
  )
110
  if uses_ple_or_qsa and (
111
+ getattr(parallel_config, "enable_dbo", False)
112
+ or getattr(parallel_config, "ubatch_size", 1) > 1
113
  ):
114
  raise NotImplementedError(
115
  "Qwen4Exp PLE/QSA does not support dual-batch overlap or microbatching"
116
  )
117
+ if getattr(text_config, "ple_layer_ids", None) and parallel_config.pipeline_parallel_size > 1:
118
  raise NotImplementedError(
119
  "Qwen4Exp N-gram PLE embedding requires pipeline_parallel_size=1 "
120
  "because non-first pipeline ranks do not receive the raw input_ids "
121
  "it needs. Please run with PP=1."
122
  )
123
  multimodal_config = vllm_config.model_config.multimodal_config
124
+ if multimodal_config is not None and getattr(multimodal_config, "language_model_only", False):
125
  _strip_qwen4_exp_mrope(vllm_config.model_config)
126
  spec_config = vllm_config.speculative_config
127
  if spec_config is not None:
 
150
  )
151
  _strip_qwen4_exp_mrope(vllm_config.model_config)
152
 
153
+ supported = set(ModelRegistry.get_supported_archs())
154
  for arch in ARCHS:
155
+ if arch not in supported:
156
  ModelRegistry.register_model(arch, f"pb27_vllm.model:{arch}")
157
  for arch in LLAMA_ARCHS:
158
+ if arch not in supported:
159
  ModelRegistry.register_model(arch, f"pb27_vllm.llama_model:{arch}")
160
  for arch in QWEN4_EXP_ARCHS:
161
+ if arch not in supported:
162
  ModelRegistry.register_model(arch, f"pb27_vllm.qwen4_exp_model:{arch}")
163
  for arch in UNIVERSAL_ARCHS:
164
+ if arch not in supported:
165
  ModelRegistry.register_model(arch, f"pb27_vllm.universal_models:{arch}")
166
 
167
  # The verify-and-update hooks of the stock Qwen3.5 classes (mamba SSM cache
168
  # dtype from the HF config; the CausalLM variant also drops the M-RoPE
169
  # fields). PB-27 keeps model_type qwen3_5(_moe) / qwen3_5(_moe)_text so they
170
+ # apply as is; the stock MoE classes map to the same hooks.
171
  for arch in ("PB27ForConditionalGeneration", "PB27MoeForConditionalGeneration"):
172
  MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForConditionalGenerationConfig)
173
  for arch in ("PB27ForCausalLM", "PB27MoeForCausalLM"):
174
  MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForCausalLMConfig)
 
175
  for arch in (
176
  *LLAMA_ARCHS,
177
  "PB27Qwen3NextForCausalLM",
 
190
  PB27Qwen4ExpForCausalLMConfig,
191
  )
192
 
193
+ if vllm_engram is not None and hasattr(vllm_engram, "_NGRAM_LAYER_FIELDS"):
194
+ for arch in QWEN4_EXP_ARCHS:
195
+ vllm_engram._NGRAM_LAYER_FIELDS[arch] = "ple_layer_ids"
196
+ if vllm_cfg is not None and hasattr(vllm_cfg, "DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES"):
197
+ vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES = (
198
+ vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES | frozenset(QWEN4_EXP_ARCHS)
199
+ )
200
+ if hasattr(vllm_cfg, "default_breakable_cudagraph_architectures") and hasattr(
201
+ vllm_cfg.default_breakable_cudagraph_architectures, "cache_clear"
202
+ ):
203
+ vllm_cfg.default_breakable_cudagraph_architectures.cache_clear()
204
 
205
  # Exact memory tier (PB27_MEMORY env): archive bookkeeping hooks on the GPU model runner.
206
  from . import memory
 
210
  try:
211
  memory.install_runner_hooks()
212
  except Exception as exc: # the API-server process has no runner; the engine/worker processes do
213
+ log.info("pb27 memory hooks not installed here: %s", exc)
 
 
pb27_vllm_plugin/pb27_vllm/memory.py CHANGED
@@ -77,6 +77,12 @@ DECODE_STRIDE = int(os.environ.get("PB27_MEMORY_DECODE_STRIDE", "64") or 64)
77
  DECODE_BLOCKS = int(os.environ.get("PB27_MEMORY_DECODE_BLOCKS", "64") or 64)
78
  DECODE_RERANK = os.environ.get("PB27_MEMORY_DECODE_RERANK", "0") == "1"
79
  SBLK = int(os.environ.get("PB27_MEMORY_SUPERBLOCK", "64") or 64)
 
 
 
 
 
 
80
  LAYER_STRIDE = int(os.environ.get("PB27_MEMORY_LAYER_STRIDE", "1") or 1)
81
  GPU_ARC_CAP = int(os.environ.get("PB27_MEMORY_GPU_ARC_CAP", "262144") or 262144)
82
  QBUF_CAP = 256
@@ -88,7 +94,7 @@ PROMPT_LENS: dict[str, int] = globals().get("PROMPT_LENS", {}) # req_i
88
  ARCHIVES: dict[tuple[str, str], "Archive"] = globals().get("ARCHIVES", {}) # (req_id, layer_name) -> Archive
89
  STATS: dict = globals().get("STATS", {"calls": 0, "fallback": 0, "retrievals": 0, "pieces": 0, "tokens_archived": 0,
90
  "adopt_full": 0, "adopt_partial": 0, "adopt_none": 0, "hole_skips": 0,
91
- "pertoken": 0, "pertoken_capped": 0})
92
  # Prefix caching: vLLM skips computing the cached prefix of a new request, so the archive of the request that
93
  # produced that prefix is retained after it finishes and adopted by the next request whose tokens match it.
94
  RETAIN = int(os.environ.get("PB27_MEMORY_RETAIN", "0") or 0) # max retired archive sets kept (0 = off)
@@ -147,6 +153,12 @@ class Archive:
147
  cached_cu: tuple | None = None # (t, w_len, p_len, cu_q, cu_w, cu_p) cached int32 cu_seqlens on GPU
148
  smax: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block max
149
  smin: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block min
 
 
 
 
 
 
150
  mapped_files: list[tuple[int, mmap.mmap, int]] = field(default_factory=list)
151
  last_evict_pos: int = 0
152
 
@@ -176,6 +188,10 @@ class Archive:
176
  mapped=True,
177
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
178
  smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
 
 
 
 
179
  mapped_files=[mf_k, mf_v])
180
  if OFFLOAD == "nvme":
181
  cap = min(max(cap, -(-max_hint // block) * block), gpu_cap)
@@ -187,7 +203,11 @@ class Archive:
187
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
188
  mapped=False,
189
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
190
- smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device))
 
 
 
 
191
  kv_dev = torch.device("cpu") if "cpu" in OFFLOAD else device
192
  kv_dtype = torch.float8_e4m3fn if "fp8" in OFFLOAD else dtype
193
  n_blk = cap // block
@@ -197,7 +217,11 @@ class Archive:
197
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
198
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
199
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
200
- smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device))
 
 
 
 
201
 
202
  def grow(self, need: int, block: int) -> None:
203
  cap = self.kmax.shape[1] * block if self.mapped else self.keys.shape[1]
@@ -236,16 +260,21 @@ class Archive:
236
  buf[:, :cap] = old
237
  setattr(self, name, buf)
238
  new_n_blk = new_cap // block
239
- for name in ("kmax", "kmin"):
240
- old = getattr(self, name)
241
- buf = torch.empty(kv, new_n_blk, d, dtype=old.dtype, device=old.device)
242
- buf[:, : cap // block] = old
243
- setattr(self, name, buf)
 
 
 
 
 
244
  new_n_sb = (new_n_blk + SBLK - 1) // SBLK
245
- for name in ("smax", "smin"):
246
  old = getattr(self, name, None)
247
  if old is not None and old.shape[1] < new_n_sb:
248
- buf = torch.empty(kv, new_n_sb, d, dtype=old.dtype, device=old.device)
249
  buf[:, : old.shape[1]] = old
250
  setattr(self, name, buf)
251
 
@@ -264,11 +293,19 @@ class Archive:
264
  n_full = full_sb_end - sb0
265
  self.smax[:, sb0:full_sb_end] = self.kmax[:, b0:b1].view(kv, n_full, SBLK, d).amax(2)
266
  self.smin[:, sb0:full_sb_end] = self.kmin[:, b0:b1].view(kv, n_full, SBLK, d).amin(2)
 
 
 
 
267
  for sb in range(max(sb0, full_sb_end), sb1 + 1):
268
  b0, b1 = sb * SBLK, min(n_blk, (sb + 1) * SBLK)
269
  if b1 > b0:
270
  self.smax[:, sb] = self.kmax[:, b0:b1].amax(1)
271
  self.smin[:, sb] = self.kmin[:, b0:b1].amin(1)
 
 
 
 
272
 
273
  def write(self, pos: torch.Tensor, k: torch.Tensor, v: torch.Tensor, block: int,
274
  p0: int | None = None, hi: int | None = None) -> None:
@@ -315,8 +352,13 @@ class Archive:
315
  s0 = ORIGIN + b_start * block - p0
316
  s1 = ORIGIN + b_end * block - p0
317
  k_full = kt[:, s0:s1].view(kt.shape[0], b_end - b_start, block, kt.shape[2])
 
318
  self.kmax[:, b_start:b_end] = k_full.amax(2).to(self.kmax.dtype)
319
  self.kmin[:, b_start:b_end] = k_full.amin(2).to(self.kmin.dtype)
 
 
 
 
320
  fast_bounds = True
321
  if contig:
322
  if self.keys.device.type == "cpu" and kt.is_cuda and self.keys.dtype == kt.dtype and n <= 8192:
@@ -362,14 +404,23 @@ class Archive:
362
  return
363
  if self.gpu_k is not None and lo >= self.gpu_lo and hi <= self.gpu_hi:
364
  seg = self.gpu_k[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float()
 
365
  else:
366
  seg = self.keys[:, lo:hi].to(device=self.kmax.device).float()
 
367
  pad = (-seg.shape[1]) % block
368
  if pad:
369
  seg = torch.cat((seg, seg[:, -1:].expand(-1, pad, -1)), dim=1)
 
 
370
  blocks = seg.view(seg.shape[0], -1, block, seg.shape[2])
371
  self.kmax[:, lo_b: lo_b + blocks.shape[1]] = blocks.amax(2).to(self.kmax.dtype)
372
  self.kmin[:, lo_b: lo_b + blocks.shape[1]] = blocks.amin(2).to(self.kmin.dtype)
 
 
 
 
 
373
 
374
  def refresh_bounds(self, block: int) -> None:
375
  if self.dirty_hi < self.dirty_lo:
@@ -520,9 +571,15 @@ def _get_rope_inv_freq(device: torch.device, d_rot: int) -> torch.Tensor:
520
  return inv_freq
521
 
522
 
523
- def _reposition_rope_sinks(sk: torch.Tensor, ref_pos: int, span: int | None = None) -> torch.Tensor:
524
- """Reposition the sink keys `sk` [KV, ns, D] from original positions `0..ns-1` to sit immediately before the
525
- repositioned archive span `[ref_pos - ROPE_ARC_SPAN, ref_pos)` when `ref_pos > ROPE_HORIZON`."""
 
 
 
 
 
 
526
  if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or sk.numel() == 0:
527
  return sk
528
  kv, ns, d = sk.shape
@@ -530,8 +587,10 @@ def _reposition_rope_sinks(sk: torch.Tensor, ref_pos: int, span: int | None = No
530
  half = d_rot // 2
531
  if half <= 0:
532
  return sk
533
- arc_span = ROPE_ARC_SPAN if span is None else span
534
- delta_val = float(max(0, ref_pos - arc_span - ORIGIN))
 
 
535
  if delta_val <= 0.0:
536
  return sk
537
  inv_freq = _get_rope_inv_freq(sk.device, d_rot)
@@ -557,12 +616,17 @@ def _reposition_rope_blocks(
557
  span: int | None = None,
558
  prio_rank: torch.Tensor | None = None,
559
  ) -> torch.Tensor:
560
- """Compress archived-block rotary positions into `[ref_pos - ROPE_ARC_SPAN, ref_pos)` when `ref_pos > ROPE_HORIZON`.
561
-
562
- Because 2D Neox-style RoPE is an abelian group (`RoPE(RoPE(k, p), delta) = RoPE(k, p + delta)`), rotating all
563
- `block` tokens of block `b` by the same shift `delta_b` preserves exact `+1` relative token offsets inside each
564
- block, and assigning the same `delta_b` across contiguous block runs `[b, b + 1, ...]` preserves exact `+1`
565
- relative token offsets across block boundaries without cross-head rotary drift.
 
 
 
 
 
566
  """
567
  if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or rk.numel() == 0:
568
  return rk
@@ -575,28 +639,29 @@ def _reposition_rope_blocks(
575
  if half <= 0:
576
  return rk
577
  idx = blk_idx.to(device=rk.device, dtype=torch.long)
578
- arc_span = ROPE_ARC_SPAN if span is None else span
579
- if contiguous_runs and arc_span > 0 and n_blk > 1:
580
- is_succ = torch.zeros_like(idx, dtype=torch.bool)
581
- is_succ[:, 1:] = idx[:, 1:] == (idx[:, :-1] + 1)
582
- if prio_rank is not None and prio_rank.shape == idx.shape:
583
- pr = prio_rank.to(device=idx.device)
584
- is_succ[:, 1:] = is_succ[:, 1:] & ((pr[:, 1:] > pr[:, :-1]) | (pr[:, :-1] < 128))
585
- step = 1
586
- while step < n_blk:
587
- prev_idx = torch.cat((idx[:, :step], idx[:, :-step]), dim=1)
588
- prev_succ = torch.cat((torch.zeros_like(is_succ[:, :step]), is_succ[:, :-step]), dim=1)
589
- idx = torch.where(is_succ, prev_idx, idx)
590
- is_succ = is_succ & prev_succ
591
- step <<= 1
592
  orig_pos = (ORIGIN + idx * block).to(torch.float32)
593
- if arc_span == 0 or not contiguous_runs:
594
- end_pos = float(max(0, ref_pos - 2 * block))
595
- delta = (end_pos - orig_pos).clamp(min=0.0)
596
- else:
597
  scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
598
  target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
599
  delta = (target_pos - orig_pos).clamp(min=0.0)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
600
  inv_freq = _get_rope_inv_freq(rk.device, d_rot)
601
  angles = delta.unsqueeze(-1) * inv_freq
602
  cos = angles.cos().unsqueeze(2)
@@ -612,6 +677,40 @@ def _reposition_rope_blocks(
612
  return out.view(kv, total_t, d)
613
 
614
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
615
  def _gather_blocks(t: torch.Tensor, blk_idx: torch.Tensor, block: int) -> torch.Tensor:
616
  """Gather contiguous blocks `blk_idx` [KV, B] starting at ORIGIN from `t` [KV, cap, D] -> [KV, B * block, D]."""
617
  kv, cap, d = t.shape
@@ -861,18 +960,66 @@ def _second_hop_blocks(arc: Archive, blk_ord: torch.Tensor, full: int, n_valid:
861
  return cand2.gather(-1, keep2)
862
 
863
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
864
  def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit: int, budget: int = 0,
865
  lead: int | None = None, union: int = 0,
866
  q_hist: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor | None] | None:
867
- """Per-query Z-score + tiered + 2-hop retrieval with [b, b+1] successor expansion and prompt-question retention."""
 
868
  block = MEMORY["block_size"]
869
  full = (limit - ORIGIN) // block
870
  if full <= 0:
871
  return None
 
872
  if lead == 0 and q.shape[0] <= MEMORY["chunk"]:
873
  arc.decode_step += 1
874
- if (DECODE_STRIDE > 1 and arc.cached_top is not None
875
- and arc.decode_step % DECODE_STRIDE != 0):
876
  return arc.cached_top, None
877
  elif lead is not None and lead < 0:
878
  arc.decode_step = 0
@@ -880,6 +1027,8 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
880
  arc.cached_top = arc.cached_rk = arc.cached_rv = None
881
  arc.cached_pre_k = arc.cached_pre_v = None
882
  arc.cached_ns = -1
 
 
883
  arc.refresh_bounds(block)
884
  q_step = q[:1] if (lead == 0 and q.shape[0] > 1) else q
885
  q_eval = (torch.cat((q_hist[:-1], q_step), dim=0)
@@ -905,7 +1054,7 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
905
  if lead == 0 and not DECODE_RERANK:
906
  k_dec = min(DECODE_BLOCKS if DECODE_BLOCKS > 0 else (MEMORY["decode_blocks"] or MEMORY["blocks"]), n_valid)
907
  fine_idx = None
908
- if SBLK > 1 and arc.smax is not None and arc.smin is not None and full > SBLK * 16:
909
  n_sb = (full + SBLK - 1) // SBLK
910
  smax_t = arc.smax[:, :n_sb].to(s_dtype).transpose(-1, -2)
911
  smin_t = arc.smin[:, :n_sb].to(s_dtype).transpose(-1, -2)
@@ -937,6 +1086,10 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
937
  s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
938
  z_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
939
  z_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
 
 
 
 
940
  ranks = _ranks(score)
941
  r_tier = ranks.amin(1) * n_pool + _ranks(score.amax(1))
942
  if t_ev > 8:
@@ -956,7 +1109,11 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
956
  (-r_tier).topk(k_sel, dim=-1).indices), dim=-1).view(kv, 3 * k_sel)
957
  keep_ord = _dedup_priority(keep_inter, k_sel)
958
  blk_ord = fine_idx.gather(-1, keep_ord) if fine_idx is not None else keep_ord
959
- if arc.decode_top is not None:
 
 
 
 
960
  d_prev = arc.decode_top.clamp(max=full - 1)
961
  k_d = min(d_prev.shape[1], blk_ord.shape[1])
962
  d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
@@ -969,13 +1126,15 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
969
  if arc.prompt_top is not None:
970
  p_prev = arc.prompt_top.clamp(max=full - 1)
971
  k_out = min(p_prev.shape[1], n_valid)
972
- n_pin = max(0, k_out - dec_prio.shape[1])
973
  merged = torch.cat((p_prev[:, :n_pin], dec_prio, p_prev[:, n_pin:]), dim=-1)
974
  top_prio = _dedup_priority(merged, k_out)
975
  blk, arc.last_prio_rank = top_prio.sort(-1)
976
  else:
977
  _, arc.last_prio_rank = dec_prio.sort(-1)
978
  blk = dec_sorted
 
 
979
  STATS["retrievals"] += 1
980
  STATS["pertoken"] += 1
981
  return blk, None
@@ -1008,6 +1167,14 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
1008
  s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
1009
  z_score_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
1010
  z_score_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
 
 
 
 
 
 
 
 
1011
  ranks = _ranks(score) # [KV, T_eval, full]
1012
  r_tier = ranks.amin(1) * full + _ranks(score.amax(1)) # [KV, full]
1013
  if lead is not None and lead < 0 and t_ev > 36:
@@ -1082,6 +1249,10 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
1082
  (-e_tier).topk(k, dim=-1).indices), dim=-1).view(kv, 3 * k)
1083
  keep_ord = _dedup_priority(keep_inter, k) # [KV, k]
1084
  blk_ord = cand.gather(-1, keep_ord) # [KV, k]
 
 
 
 
1085
  if MEMORY["rerank"] and n_valid > 16:
1086
  n_seed = min(8, blk_ord.shape[1])
1087
  cur_head = blk_ord[:, :n_seed]
@@ -1097,7 +1268,7 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
1097
  arc.decode_top = None
1098
  merged_ord = blk_ord
1099
  elif arc.prompt_top is not None:
1100
- if arc.decode_top is not None:
1101
  d_prev = arc.decode_top.clamp(max=full - 1)
1102
  k_d = min(d_prev.shape[1], blk_ord.shape[1])
1103
  d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
@@ -1116,6 +1287,8 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
1116
  blk, arc.last_prio_rank = top_prio.sort(-1)
1117
  if lead is not None and lead < 0:
1118
  arc.prompt_top = top_prio
 
 
1119
  STATS["retrievals"] += 1
1120
  STATS["pertoken"] += 1
1121
  return blk, None
@@ -1139,8 +1312,8 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1139
  window: int, scale: float, own: torch.Tensor | None = None,
1140
  q_min: int | None = None, q_max: int | None = None) -> torch.Tensor:
1141
  """Exact attention of piece q [T, H, D] at positions q_pos over its sliding window (from the archive) plus the
1142
- retrieved blocks `top` [KV, k]; with `own` [KV, T, k] each query sees only its own retrieved blocks.
1143
- Returns [T, H, D]."""
1144
  block = MEMORY["block_size"]
1145
  t, h, d = q.shape
1146
  kv = h // groups
@@ -1156,6 +1329,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1156
  fold_tail = bool(use_lse and t <= MEMORY["chunk"] and FAITHFUL and top is not None and lo > tail0)
1157
  w_lo = tail0 if fold_tail else lo
1158
  ns = min(SINKS, lo) # sink positions below the window
 
1159
  if use_lse:
1160
  if arc.gpu_k is not None and w_lo >= arc.gpu_lo and hi <= arc.gpu_hi:
1161
  wk_t = arc.gpu_k[w_lo - arc.gpu_lo : hi - arc.gpu_lo].to(dtype=q.dtype) # [Wn, KV, D] zero-copy slice
@@ -1177,6 +1351,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1177
  arc.cached_rv = rv
1178
  ref_pos = lo
1179
  rk = _reposition_rope_blocks(rk_raw, top, block, ref_pos, contiguous_runs=True, prio_rank=arc.last_prio_rank)
 
1180
  if ns > 0:
1181
  if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
1182
  sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
@@ -1184,12 +1359,18 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1184
  else:
1185
  sk_kv = arc.keys[:, :ns].to(device=q.device, dtype=q.dtype)
1186
  sv_t = arc.values[:, :ns].to(device=q.device, dtype=q.dtype).transpose(0, 1)
1187
- sk_t = _reposition_rope_sinks(sk_kv, ref_pos).transpose(0, 1)
1188
- pre_k = torch.cat((sk_t, rk.transpose(0, 1)), dim=0)
1189
- pre_v = torch.cat((sv_t, rv.transpose(0, 1)), dim=0)
1190
- else:
1191
- pre_k = rk.transpose(0, 1).contiguous()
1192
- pre_v = rv.transpose(0, 1).contiguous()
 
 
 
 
 
 
1193
  arc.cached_ns = ns
1194
  arc.cached_pre_k = pre_k
1195
  arc.cached_pre_v = pre_v
@@ -1208,7 +1389,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1208
  return torch.lerp(out_w.float(), out_p.float(), w).to(q.dtype)
1209
  elif arc.gpu_k is not None and lo >= arc.gpu_lo and hi <= arc.gpu_hi:
1210
  wk = arc.gpu_k[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
1211
- wv = arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
1212
  else:
1213
  wk = arc.keys[:, lo:hi].to(device=q.device).to(dtype=q.dtype) # [KV, Wn, D]
1214
  wv = arc.values[:, lo:hi].to(device=q.device).to(dtype=q.dtype)
@@ -1216,6 +1397,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1216
  need_mask = not (use_lse or unmasked)
1217
  parts_k, parts_v, parts_allow = [], [], []
1218
  ref_pos = lo
 
1219
  if ns > 0:
1220
  if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
1221
  sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
@@ -1223,9 +1405,16 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1223
  else:
1224
  sk_kv = arc.keys[:, :ns].to(device=q.device).to(dtype=q.dtype)
1225
  parts_v.append(arc.values[:, :ns].to(device=q.device).to(dtype=q.dtype))
1226
- parts_k.append(_reposition_rope_sinks(sk_kv, ref_pos))
1227
  if need_mask:
1228
  parts_allow.append(torch.ones(t, ns, dtype=torch.bool, device=q.device))
 
 
 
 
 
 
 
1229
  if top is not None:
1230
  if arc.last_top is top and arc.last_rk is not None:
1231
  rk_raw = arc.last_rk.to(dtype=q.dtype)
@@ -1301,52 +1490,118 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
1301
  return out[0].reshape(h, t, d).transpose(0, 1).to(q.dtype)
1302
 
1303
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1304
  def install_runner_hooks() -> None:
1305
- """Expose the runner's current InputBatch and free archives of finished requests (V2 GPU runner)."""
1306
- from vllm.v1.worker.gpu.model_runner import GPUModelRunner
1307
- from vllm.v1.worker.gpu.states import RequestState
 
 
1308
 
1309
- if getattr(GPUModelRunner, "_pb27_hooked", False):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1310
  return
1311
- orig_prepare = GPUModelRunner.prepare_inputs
1312
- orig_add = RequestState.add_request
1313
- orig_remove = RequestState.remove_request
1314
-
1315
- def prepare_inputs(self, *args, **kwargs):
1316
- ib = orig_prepare(self, *args, **kwargs)
1317
- BATCH["ib"] = ib
1318
- BATCH["pos_list"] = None
1319
- BATCH["req_bounds"] = None
1320
- return ib
1321
-
1322
- def add_request(self, req_id, prompt_len, *args, **kwargs):
1323
- PROMPT_LENS[req_id] = int(prompt_len)
1324
- all_token_ids = kwargs.get("all_token_ids", args[0] if len(args) > 0 else None)
1325
- num_computed = kwargs.get("num_computed_tokens", args[1] if len(args) > 1 else 0)
1326
- if all_token_ids is not None and num_computed:
1327
- adopt_for_request(req_id, all_token_ids, int(num_computed))
1328
- if int(prompt_len) > 262144 and not ADOPTED.get(req_id, ({},))[0] and len(RETIRED) >= max(1, RETAIN):
1329
- while RETIRED:
1330
- _free(RETIRED.pop(0)["archives"])
1331
- return orig_add(self, req_id, prompt_len, *args, **kwargs)
1332
-
1333
- def remove_request(self, req_id, *args, **kwargs):
1334
- tokens = None
1335
- if RETAIN > 0 and req_id in self.req_id_to_index:
1336
- idx = self.req_id_to_index[req_id]
1337
- try: # prompt + accepted output tokens, maintained on the GPU by the runner's post-update step
1338
- n = int(self.total_len.gpu[idx])
1339
- tokens = self.all_token_ids.gpu[idx, :n].clone()
1340
- except Exception as exc: # internals differ: fall back to no retention
1341
- log.warning("pb27 memory: cannot read token ids for %s (%s); not retaining", req_id, exc)
1342
- drop_request(req_id, tokens)
1343
- return orig_remove(self, req_id, *args, **kwargs)
1344
-
1345
- GPUModelRunner.prepare_inputs = prepare_inputs
1346
- RequestState.add_request = add_request
1347
- RequestState.remove_request = remove_request
1348
- GPUModelRunner._pb27_hooked = True
1349
- log.info("pb27 memory tier: runner hooks installed (pid %d), config %s", os.getpid(), MEMORY)
1350
 
1351
 
1352
  def _layer_enabled(layer_name: str) -> bool:
@@ -1391,9 +1646,11 @@ def windowed_attention_with_memory(attn_module, q: torch.Tensor, k: torch.Tensor
1391
  STATS["calls"] += 1
1392
  if DEBUG and STATS["calls"] % 500 == 1:
1393
  log.info("pb27 memory: call %d ib=%s archives=%d stats=%s", STATS["calls"], ib is not None, len(ARCHIVES), STATS)
1394
- if ib is None or not ib.req_ids:
1395
- STATS["fallback"] += 1
1396
- return attn_module(q, k, v)
 
 
1397
  layer = attn_module.layer_name
1398
  if not _layer_enabled(layer):
1399
  return attn_module(q, k, v)
@@ -1538,15 +1795,16 @@ def windowed_attention_with_memory(attn_module, q: torch.Tensor, k: torch.Tensor
1538
  budget = 0
1539
  rr = (p1 - p0) <= q_piece if PREFILL == "pt" else rerank
1540
  q_hist = None
 
1541
  need_q_hist = (
1542
  (QWIN or PERTOKEN)
1543
  and not is_prefill
1544
  and arc.qbuf is not None
1545
  and (
1546
  QWIN
1547
- or DECODE_STRIDE <= 1
1548
  or arc.cached_top is None
1549
- or (arc.decode_step + 1) % DECODE_STRIDE == 0
1550
  )
1551
  )
1552
  if need_q_hist:
 
77
  DECODE_BLOCKS = int(os.environ.get("PB27_MEMORY_DECODE_BLOCKS", "64") or 64)
78
  DECODE_RERANK = os.environ.get("PB27_MEMORY_DECODE_RERANK", "0") == "1"
79
  SBLK = int(os.environ.get("PB27_MEMORY_SUPERBLOCK", "64") or 64)
80
+ # Dense Global Aggregation ("auto", "1", or "0"): detects high-entropy global queries across the archive and activates
81
+ # (1) stratified super-block timeline coverage, (2) novelty-rotating decode working sets across decode steps, and
82
+ # (3) resident super-block summary KV landmarks (`smean_k` / `smean_v`, 1 summary token per 4,096 tokens).
83
+ GLOBAL_AGG = os.environ.get("PB27_MEMORY_GLOBAL_AGG", "auto").strip().lower()
84
+ GLOBAL_DECODE_STRIDE = int(os.environ.get("PB27_MEMORY_GLOBAL_DECODE_STRIDE", "16") or 16)
85
+ NOVELTY_ALPHA = float(os.environ.get("PB27_MEMORY_NOVELTY_ALPHA", "0.75") or 0.75)
86
  LAYER_STRIDE = int(os.environ.get("PB27_MEMORY_LAYER_STRIDE", "1") or 1)
87
  GPU_ARC_CAP = int(os.environ.get("PB27_MEMORY_GPU_ARC_CAP", "262144") or 262144)
88
  QBUF_CAP = 256
 
94
  ARCHIVES: dict[tuple[str, str], "Archive"] = globals().get("ARCHIVES", {}) # (req_id, layer_name) -> Archive
95
  STATS: dict = globals().get("STATS", {"calls": 0, "fallback": 0, "retrievals": 0, "pieces": 0, "tokens_archived": 0,
96
  "adopt_full": 0, "adopt_partial": 0, "adopt_none": 0, "hole_skips": 0,
97
+ "pertoken": 0, "pertoken_capped": 0, "global_agg": 0})
98
  # Prefix caching: vLLM skips computing the cached prefix of a new request, so the archive of the request that
99
  # produced that prefix is retained after it finishes and adopted by the next request whose tokens match it.
100
  RETAIN = int(os.environ.get("PB27_MEMORY_RETAIN", "0") or 0) # max retired archive sets kept (0 = off)
 
153
  cached_cu: tuple | None = None # (t, w_len, p_len, cu_q, cu_w, cu_p) cached int32 cu_seqlens on GPU
154
  smax: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block max
155
  smin: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block min
156
+ bmean_k: torch.Tensor | None = None # [KV, cap // block, D] per-block mean key on GPU for hierarchical summaries
157
+ bmean_v: torch.Tensor | None = None # [KV, cap // block, D] per-block mean value on GPU for hierarchical summaries
158
+ smean_k: torch.Tensor | None = None # [KV, n_sb, D] coarse super-block summary key landmarks on GPU
159
+ smean_v: torch.Tensor | None = None # [KV, n_sb, D] coarse super-block summary value landmarks on GPU
160
+ visit_counts: torch.Tensor | None = None # [KV, cap // block] int16 decode visit counts for novelty sweeping
161
+ is_global_agg: bool = False # True when Dense Global Aggregation mode is active for this request
162
  mapped_files: list[tuple[int, mmap.mmap, int]] = field(default_factory=list)
163
  last_evict_pos: int = 0
164
 
 
188
  mapped=True,
189
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
190
  smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
191
+ bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
192
+ bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
193
+ smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
194
+ smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
195
  mapped_files=[mf_k, mf_v])
196
  if OFFLOAD == "nvme":
197
  cap = min(max(cap, -(-max_hint // block) * block), gpu_cap)
 
203
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
204
  mapped=False,
205
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
206
+ smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
207
+ bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
208
+ bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
209
+ smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
210
+ smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device))
211
  kv_dev = torch.device("cpu") if "cpu" in OFFLOAD else device
212
  kv_dtype = torch.float8_e4m3fn if "fp8" in OFFLOAD else dtype
213
  n_blk = cap // block
 
217
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
218
  torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
219
  smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
220
+ smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
221
+ bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
222
+ bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
223
+ smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
224
+ smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device))
225
 
226
  def grow(self, need: int, block: int) -> None:
227
  cap = self.kmax.shape[1] * block if self.mapped else self.keys.shape[1]
 
260
  buf[:, :cap] = old
261
  setattr(self, name, buf)
262
  new_n_blk = new_cap // block
263
+ for name in ("kmax", "kmin", "bmean_k", "bmean_v"):
264
+ old = getattr(self, name, None)
265
+ if old is not None:
266
+ buf = torch.zeros(kv, new_n_blk, d, dtype=old.dtype, device=old.device)
267
+ buf[:, : old.shape[1]] = old
268
+ setattr(self, name, buf)
269
+ if self.visit_counts is not None and self.visit_counts.shape[1] < new_n_blk:
270
+ vc = torch.zeros(kv, new_n_blk, dtype=self.visit_counts.dtype, device=self.visit_counts.device)
271
+ vc[:, : self.visit_counts.shape[1]] = self.visit_counts
272
+ self.visit_counts = vc
273
  new_n_sb = (new_n_blk + SBLK - 1) // SBLK
274
+ for name in ("smax", "smin", "smean_k", "smean_v"):
275
  old = getattr(self, name, None)
276
  if old is not None and old.shape[1] < new_n_sb:
277
+ buf = torch.zeros(kv, new_n_sb, d, dtype=old.dtype, device=old.device)
278
  buf[:, : old.shape[1]] = old
279
  setattr(self, name, buf)
280
 
 
293
  n_full = full_sb_end - sb0
294
  self.smax[:, sb0:full_sb_end] = self.kmax[:, b0:b1].view(kv, n_full, SBLK, d).amax(2)
295
  self.smin[:, sb0:full_sb_end] = self.kmin[:, b0:b1].view(kv, n_full, SBLK, d).amin(2)
296
+ if self.bmean_k is not None and self.smean_k is not None:
297
+ self.smean_k[:, sb0:full_sb_end] = self.bmean_k[:, b0:b1].view(kv, n_full, SBLK, d).mean(2).to(self.smean_k.dtype)
298
+ if self.bmean_v is not None and self.smean_v is not None:
299
+ self.smean_v[:, sb0:full_sb_end] = self.bmean_v[:, b0:b1].view(kv, n_full, SBLK, d).mean(2).to(self.smean_v.dtype)
300
  for sb in range(max(sb0, full_sb_end), sb1 + 1):
301
  b0, b1 = sb * SBLK, min(n_blk, (sb + 1) * SBLK)
302
  if b1 > b0:
303
  self.smax[:, sb] = self.kmax[:, b0:b1].amax(1)
304
  self.smin[:, sb] = self.kmin[:, b0:b1].amin(1)
305
+ if self.bmean_k is not None and self.smean_k is not None:
306
+ self.smean_k[:, sb] = self.bmean_k[:, b0:b1].mean(1).to(self.smean_k.dtype)
307
+ if self.bmean_v is not None and self.smean_v is not None:
308
+ self.smean_v[:, sb] = self.bmean_v[:, b0:b1].mean(1).to(self.smean_v.dtype)
309
 
310
  def write(self, pos: torch.Tensor, k: torch.Tensor, v: torch.Tensor, block: int,
311
  p0: int | None = None, hi: int | None = None) -> None:
 
352
  s0 = ORIGIN + b_start * block - p0
353
  s1 = ORIGIN + b_end * block - p0
354
  k_full = kt[:, s0:s1].view(kt.shape[0], b_end - b_start, block, kt.shape[2])
355
+ v_full = vt[:, s0:s1].view(vt.shape[0], b_end - b_start, block, vt.shape[2])
356
  self.kmax[:, b_start:b_end] = k_full.amax(2).to(self.kmax.dtype)
357
  self.kmin[:, b_start:b_end] = k_full.amin(2).to(self.kmin.dtype)
358
+ if self.bmean_k is not None:
359
+ self.bmean_k[:, b_start:b_end] = k_full.mean(2).to(self.bmean_k.dtype)
360
+ if self.bmean_v is not None:
361
+ self.bmean_v[:, b_start:b_end] = v_full.mean(2).to(self.bmean_v.dtype)
362
  fast_bounds = True
363
  if contig:
364
  if self.keys.device.type == "cpu" and kt.is_cuda and self.keys.dtype == kt.dtype and n <= 8192:
 
404
  return
405
  if self.gpu_k is not None and lo >= self.gpu_lo and hi <= self.gpu_hi:
406
  seg = self.gpu_k[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float()
407
+ seg_v = self.gpu_v[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float() if self.gpu_v is not None else None
408
  else:
409
  seg = self.keys[:, lo:hi].to(device=self.kmax.device).float()
410
+ seg_v = self.values[:, lo:hi].to(device=self.kmax.device).float() if self.bmean_v is not None else None
411
  pad = (-seg.shape[1]) % block
412
  if pad:
413
  seg = torch.cat((seg, seg[:, -1:].expand(-1, pad, -1)), dim=1)
414
+ if seg_v is not None:
415
+ seg_v = torch.cat((seg_v, seg_v[:, -1:].expand(-1, pad, -1)), dim=1)
416
  blocks = seg.view(seg.shape[0], -1, block, seg.shape[2])
417
  self.kmax[:, lo_b: lo_b + blocks.shape[1]] = blocks.amax(2).to(self.kmax.dtype)
418
  self.kmin[:, lo_b: lo_b + blocks.shape[1]] = blocks.amin(2).to(self.kmin.dtype)
419
+ if self.bmean_k is not None:
420
+ self.bmean_k[:, lo_b: lo_b + blocks.shape[1]] = blocks.mean(2).to(self.bmean_k.dtype)
421
+ if self.bmean_v is not None and seg_v is not None:
422
+ blocks_v = seg_v.view(seg_v.shape[0], -1, block, seg_v.shape[2])
423
+ self.bmean_v[:, lo_b: lo_b + blocks_v.shape[1]] = blocks_v.mean(2).to(self.bmean_v.dtype)
424
 
425
  def refresh_bounds(self, block: int) -> None:
426
  if self.dirty_hi < self.dirty_lo:
 
571
  return inv_freq
572
 
573
 
574
+ def _reposition_rope_sinks(
575
+ sk: torch.Tensor,
576
+ ref_pos: int,
577
+ span: int | None = None,
578
+ n_blk: int = 0,
579
+ block: int = 64,
580
+ ) -> torch.Tensor:
581
+ """Reposition the sink keys `sk` [KV, ns, D] from original positions `0..ns-1` to `[base_start - ns, base_start)`,
582
+ strictly before the earliest repositioned archive block `[base_start, ref_pos)` when `ref_pos > ROPE_HORIZON`."""
583
  if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or sk.numel() == 0:
584
  return sk
585
  kv, ns, d = sk.shape
 
587
  half = d_rot // 2
588
  if half <= 0:
589
  return sk
590
+ arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
591
+ eff_span = max(arc_span, n_blk * block)
592
+ base_start = max(ORIGIN, ref_pos - eff_span)
593
+ delta_val = float(max(0, base_start - ns))
594
  if delta_val <= 0.0:
595
  return sk
596
  inv_freq = _get_rope_inv_freq(sk.device, d_rot)
 
616
  span: int | None = None,
617
  prio_rank: torch.Tensor | None = None,
618
  ) -> torch.Tensor:
619
+ """Piecewise-Proportional, Strictly Non-Overlapping Block RoPE into `[ref_pos - eff_span, ref_pos)`.
620
+
621
+ Guarantees three simultaneous invariants when `ref_pos > ROPE_HORIZON`:
622
+ 1. Exact Intra-Block & Contiguous-Run Continuity: All `block` tokens inside block `b` rotate by the same shift
623
+ `delta_b` (preserving exact `+1` local steps), and contiguous blocks `b_j = b_{j-1} + 1` have inter-block
624
+ historical gap `g_mid = 0`, receiving the exact same `delta` across the block boundary.
625
+ 2. Strict Chronological Non-Overlap: Every block `j` is allocated a mandatory `block`-token footprint plus its
626
+ proportional share of the remaining RoPE slack, guaranteeing `T(b_j) >= T(b_{j-1}) + block` so nearby
627
+ non-adjacent blocks (`b_j = b_{j-1} + 2`) never invert or overlap in rotary phase space.
628
+ 3. Proportional Macro-Timeline Preservation: True historical gaps `g_j = (b_j - b_{j-1} - 1) * block` scale
629
+ proportionally across the `ROPE_ARC_SPAN` horizon instead of flattening relative timeline distances.
630
  """
631
  if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or rk.numel() == 0:
632
  return rk
 
639
  if half <= 0:
640
  return rk
641
  idx = blk_idx.to(device=rk.device, dtype=torch.long)
642
+ arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
 
 
 
 
 
 
 
 
 
 
 
 
 
643
  orig_pos = (ORIGIN + idx * block).to(torch.float32)
644
+ if not contiguous_runs:
 
 
 
645
  scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
646
  target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
647
  delta = (target_pos - orig_pos).clamp(min=0.0)
648
+ else:
649
+ eff_span = max(arc_span, n_blk * block)
650
+ base_start = float(max(ORIGIN, ref_pos - eff_span))
651
+ slack = float(max(0.0, (ref_pos - base_start) - n_blk * block))
652
+ g_pre = (orig_pos[:, :1] - float(ORIGIN)).clamp(min=0.0)
653
+ if n_blk > 1:
654
+ g_mid = ((idx[:, 1:] - idx[:, :-1] - 1).clamp(min=0) * block).to(torch.float32)
655
+ g_cum = torch.cat((g_pre, g_pre + torch.cumsum(g_mid, dim=1)), dim=1)
656
+ g_mid_sum = g_mid.sum(dim=1, keepdim=True)
657
+ else:
658
+ g_cum = g_pre
659
+ g_mid_sum = torch.zeros_like(g_pre)
660
+ g_post = (float(ref_pos) - (orig_pos[:, -1:] + float(block))).clamp(min=0.0)
661
+ g_tot = (g_pre + g_mid_sum + g_post).clamp(min=1.0)
662
+ j_steps = (torch.arange(n_blk, device=rk.device, dtype=torch.float32) * float(block)).unsqueeze(0)
663
+ target_pos = base_start + j_steps + torch.round(g_cum * (slack / g_tot))
664
+ delta = (target_pos - orig_pos).clamp(min=0.0)
665
  inv_freq = _get_rope_inv_freq(rk.device, d_rot)
666
  angles = delta.unsqueeze(-1) * inv_freq
667
  cos = angles.cos().unsqueeze(2)
 
677
  return out.view(kv, total_t, d)
678
 
679
 
680
+ def _reposition_rope_landmarks(
681
+ smk: torch.Tensor,
682
+ block: int,
683
+ ref_pos: int,
684
+ span: int | None = None,
685
+ ) -> torch.Tensor:
686
+ """Reposition super-block centroid landmark keys `smk` [KV, n_sb, D] proportionally into `[ref_pos - arc_span, ref_pos)`."""
687
+ if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or smk.numel() == 0:
688
+ return smk
689
+ kv, n_sb, d = smk.shape
690
+ d_rot = d // 4
691
+ half = d_rot // 2
692
+ if half <= 0:
693
+ return smk
694
+ arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
695
+ sb_stride = float(SBLK * block)
696
+ orig_pos = float(ORIGIN) + (torch.arange(n_sb, device=smk.device, dtype=torch.float32) + 0.5) * sb_stride
697
+ scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
698
+ target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
699
+ delta = (target_pos - orig_pos).clamp(min=0.0).unsqueeze(0) # [1, n_sb]
700
+ inv_freq = _get_rope_inv_freq(smk.device, d_rot)
701
+ angles = delta.unsqueeze(-1) * inv_freq # [1, n_sb, half]
702
+ cos = angles.cos()
703
+ sin = angles.sin()
704
+ x1 = smk[..., :half].float()
705
+ x2 = smk[..., half:d_rot].float()
706
+ r1 = (x1 * cos - x2 * sin).to(smk.dtype)
707
+ r2 = (x2 * cos + x1 * sin).to(smk.dtype)
708
+ out = smk.clone()
709
+ out[..., :half] = r1
710
+ out[..., half:d_rot] = r2
711
+ return out
712
+
713
+
714
  def _gather_blocks(t: torch.Tensor, blk_idx: torch.Tensor, block: int) -> torch.Tensor:
715
  """Gather contiguous blocks `blk_idx` [KV, B] starting at ORIGIN from `t` [KV, cap, D] -> [KV, B * block, D]."""
716
  kv, cap, d = t.shape
 
960
  return cand2.gather(-1, keep2)
961
 
962
 
963
+ def _detect_global_agg(z_raw: torch.Tensor, full: int) -> bool:
964
+ """Detect Dense Global Aggregation queries (high participation ratio across the archived timeline)."""
965
+ if GLOBAL_AGG in ("0", "false", "off", "no"):
966
+ return False
967
+ if GLOBAL_AGG in ("1", "true", "on", "yes", "force"):
968
+ return True
969
+ if full < 128:
970
+ return False
971
+ z_tail = z_raw[:, -min(z_raw.shape[1], 16):].amax(1) # [KV, full]
972
+ z_1d = (z_tail - z_tail.mean(-1, keepdim=True)) / z_tail.std(-1, keepdim=True).clamp(min=1e-6)
973
+ pos_mass = (z_1d - 1.0).clamp(min=0.0)
974
+ l1 = pos_mass.sum(dim=-1)
975
+ l2_sq = (pos_mass * pos_mass).sum(dim=-1).clamp(min=1e-6)
976
+ n_eff = float((l1 * l1 / l2_sq).mean().item())
977
+ z_peak = float(z_1d.amax().item())
978
+ return n_eff > max(64.0, 0.12 * float(full)) and z_peak < 4.5
979
+
980
+
981
+ def _stratified_superblock_peaks(score_1d: torch.Tensor, full: int, max_sb: int) -> torch.Tensor:
982
+ """Pick the highest-scoring block inside each uniform timeline segment across `[0, full)` -> `[KV, n_seg]`."""
983
+ kv = score_1d.shape[0]
984
+ n_seg = min(max(1, max_sb), full // max(1, SBLK))
985
+ if n_seg <= 1:
986
+ return score_1d.argmax(dim=-1, keepdim=True)
987
+ seg_len = full // n_seg
988
+ usable = n_seg * seg_len
989
+ s_seg = score_1d[:, :usable].view(kv, n_seg, seg_len)
990
+ local_best = s_seg.argmax(dim=-1)
991
+ seg_base = (torch.arange(n_seg, device=score_1d.device, dtype=local_best.dtype) * seg_len).unsqueeze(0)
992
+ return seg_base + local_best
993
+
994
+
995
+ def _record_block_visits(arc: Archive, blk: torch.Tensor, full: int) -> None:
996
+ """Increment per-block decode visit counts for novelty-penalized global aggregation sweeps."""
997
+ kv = blk.shape[0]
998
+ cap_blk = arc.kmax.shape[1]
999
+ if arc.visit_counts is None or arc.visit_counts.shape[1] < cap_blk:
1000
+ vc = torch.zeros(kv, cap_blk, dtype=torch.int16, device=blk.device)
1001
+ if arc.visit_counts is not None:
1002
+ vc[:, : arc.visit_counts.shape[1]] = arc.visit_counts
1003
+ arc.visit_counts = vc
1004
+ b_clamped = blk.clamp(min=0, max=cap_blk - 1)
1005
+ ones = torch.ones_like(b_clamped, dtype=torch.int16)
1006
+ arc.visit_counts.scatter_add_(1, b_clamped, ones)
1007
+
1008
+
1009
  def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit: int, budget: int = 0,
1010
  lead: int | None = None, union: int = 0,
1011
  q_hist: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor | None] | None:
1012
+ """Per-query Z-score + tiered + 2-hop retrieval with [b, b+1] successor expansion, prompt-question retention,
1013
+ and Dense Global Aggregation support (stratified timeline coverage + novelty-rotating decode sweep)."""
1014
  block = MEMORY["block_size"]
1015
  full = (limit - ORIGIN) // block
1016
  if full <= 0:
1017
  return None
1018
+ eff_stride = GLOBAL_DECODE_STRIDE if arc.is_global_agg else DECODE_STRIDE
1019
  if lead == 0 and q.shape[0] <= MEMORY["chunk"]:
1020
  arc.decode_step += 1
1021
+ if (eff_stride > 1 and arc.cached_top is not None
1022
+ and arc.decode_step % eff_stride != 0):
1023
  return arc.cached_top, None
1024
  elif lead is not None and lead < 0:
1025
  arc.decode_step = 0
 
1027
  arc.cached_top = arc.cached_rk = arc.cached_rv = None
1028
  arc.cached_pre_k = arc.cached_pre_v = None
1029
  arc.cached_ns = -1
1030
+ if arc.visit_counts is not None:
1031
+ arc.visit_counts.zero_()
1032
  arc.refresh_bounds(block)
1033
  q_step = q[:1] if (lead == 0 and q.shape[0] > 1) else q
1034
  q_eval = (torch.cat((q_hist[:-1], q_step), dim=0)
 
1054
  if lead == 0 and not DECODE_RERANK:
1055
  k_dec = min(DECODE_BLOCKS if DECODE_BLOCKS > 0 else (MEMORY["decode_blocks"] or MEMORY["blocks"]), n_valid)
1056
  fine_idx = None
1057
+ if SBLK > 1 and arc.smax is not None and arc.smin is not None and full > SBLK * 16 and not arc.is_global_agg:
1058
  n_sb = (full + SBLK - 1) // SBLK
1059
  smax_t = arc.smax[:, :n_sb].to(s_dtype).transpose(-1, -2)
1060
  smin_t = arc.smin[:, :n_sb].to(s_dtype).transpose(-1, -2)
 
1086
  s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
1087
  z_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
1088
  z_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
1089
+ if arc.is_global_agg and arc.visit_counts is not None and fine_idx is None:
1090
+ vc_pen = NOVELTY_ALPHA * torch.log1p(arc.visit_counts[:, :full].float()).unsqueeze(1)
1091
+ z_raw = z_raw - vc_pen
1092
+ z_rel = z_rel - vc_pen
1093
  ranks = _ranks(score)
1094
  r_tier = ranks.amin(1) * n_pool + _ranks(score.amax(1))
1095
  if t_ev > 8:
 
1109
  (-r_tier).topk(k_sel, dim=-1).indices), dim=-1).view(kv, 3 * k_sel)
1110
  keep_ord = _dedup_priority(keep_inter, k_sel)
1111
  blk_ord = fine_idx.gather(-1, keep_ord) if fine_idx is not None else keep_ord
1112
+ if arc.is_global_agg and fine_idx is None and full >= SBLK * 2:
1113
+ strat = _stratified_superblock_peaks(z_raw.amax(1), full, max(8, k_sel // 2))
1114
+ n_top = max(1, k_sel // 2)
1115
+ blk_ord = _dedup_priority(torch.cat((blk_ord[:, :n_top], strat, blk_ord[:, n_top:]), dim=-1), k_sel)
1116
+ if arc.decode_top is not None and not arc.is_global_agg:
1117
  d_prev = arc.decode_top.clamp(max=full - 1)
1118
  k_d = min(d_prev.shape[1], blk_ord.shape[1])
1119
  d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
 
1126
  if arc.prompt_top is not None:
1127
  p_prev = arc.prompt_top.clamp(max=full - 1)
1128
  k_out = min(p_prev.shape[1], n_valid)
1129
+ n_pin = (k_out // 2) if arc.is_global_agg else max(0, k_out - dec_prio.shape[1])
1130
  merged = torch.cat((p_prev[:, :n_pin], dec_prio, p_prev[:, n_pin:]), dim=-1)
1131
  top_prio = _dedup_priority(merged, k_out)
1132
  blk, arc.last_prio_rank = top_prio.sort(-1)
1133
  else:
1134
  _, arc.last_prio_rank = dec_prio.sort(-1)
1135
  blk = dec_sorted
1136
+ if arc.is_global_agg:
1137
+ _record_block_visits(arc, dec_prio, full)
1138
  STATS["retrievals"] += 1
1139
  STATS["pertoken"] += 1
1140
  return blk, None
 
1167
  s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
1168
  z_score_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
1169
  z_score_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
1170
+ if lead is not None and lead < 0:
1171
+ arc.is_global_agg = _detect_global_agg(z_score_raw, full)
1172
+ if arc.is_global_agg:
1173
+ STATS["global_agg"] = STATS.get("global_agg", 0) + 1
1174
+ elif lead == 0 and arc.is_global_agg and arc.visit_counts is not None:
1175
+ vc_pen = NOVELTY_ALPHA * torch.log1p(arc.visit_counts[:, :full].float()).unsqueeze(1)
1176
+ z_score_raw = z_score_raw - vc_pen
1177
+ z_score_rel = z_score_rel - vc_pen
1178
  ranks = _ranks(score) # [KV, T_eval, full]
1179
  r_tier = ranks.amin(1) * full + _ranks(score.amax(1)) # [KV, full]
1180
  if lead is not None and lead < 0 and t_ev > 36:
 
1249
  (-e_tier).topk(k, dim=-1).indices), dim=-1).view(kv, 3 * k)
1250
  keep_ord = _dedup_priority(keep_inter, k) # [KV, k]
1251
  blk_ord = cand.gather(-1, keep_ord) # [KV, k]
1252
+ if arc.is_global_agg and full >= SBLK * 2:
1253
+ strat = _stratified_superblock_peaks(z_score_raw.amax(1), full, max(16, k // 4))
1254
+ n_keep_top = max(1, (3 * k) // 4)
1255
+ blk_ord = _dedup_priority(torch.cat((blk_ord[:, :n_keep_top], strat, blk_ord[:, n_keep_top:]), dim=-1), k)
1256
  if MEMORY["rerank"] and n_valid > 16:
1257
  n_seed = min(8, blk_ord.shape[1])
1258
  cur_head = blk_ord[:, :n_seed]
 
1268
  arc.decode_top = None
1269
  merged_ord = blk_ord
1270
  elif arc.prompt_top is not None:
1271
+ if arc.decode_top is not None and not arc.is_global_agg:
1272
  d_prev = arc.decode_top.clamp(max=full - 1)
1273
  k_d = min(d_prev.shape[1], blk_ord.shape[1])
1274
  d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
 
1287
  blk, arc.last_prio_rank = top_prio.sort(-1)
1288
  if lead is not None and lead < 0:
1289
  arc.prompt_top = top_prio
1290
+ if arc.is_global_agg:
1291
+ _record_block_visits(arc, top_prio, full)
1292
  STATS["retrievals"] += 1
1293
  STATS["pertoken"] += 1
1294
  return blk, None
 
1312
  window: int, scale: float, own: torch.Tensor | None = None,
1313
  q_min: int | None = None, q_max: int | None = None) -> torch.Tensor:
1314
  """Exact attention of piece q [T, H, D] at positions q_pos over its sliding window (from the archive) plus the
1315
+ retrieved blocks `top` [KV, k] (and super-block summary landmarks when `arc.is_global_agg` is active);
1316
+ with `own` [KV, T, k] each query sees only its own retrieved blocks. Returns [T, H, D]."""
1317
  block = MEMORY["block_size"]
1318
  t, h, d = q.shape
1319
  kv = h // groups
 
1329
  fold_tail = bool(use_lse and t <= MEMORY["chunk"] and FAITHFUL and top is not None and lo > tail0)
1330
  w_lo = tail0 if fold_tail else lo
1331
  ns = min(SINKS, lo) # sink positions below the window
1332
+ n_sb_valid = max(0, ((lo - ORIGIN) // block) // SBLK) if (arc.is_global_agg and arc.smean_k is not None and arc.smean_v is not None) else 0
1333
  if use_lse:
1334
  if arc.gpu_k is not None and w_lo >= arc.gpu_lo and hi <= arc.gpu_hi:
1335
  wk_t = arc.gpu_k[w_lo - arc.gpu_lo : hi - arc.gpu_lo].to(dtype=q.dtype) # [Wn, KV, D] zero-copy slice
 
1351
  arc.cached_rv = rv
1352
  ref_pos = lo
1353
  rk = _reposition_rope_blocks(rk_raw, top, block, ref_pos, contiguous_runs=True, prio_rank=arc.last_prio_rank)
1354
+ pre_k_list, pre_v_list = [], []
1355
  if ns > 0:
1356
  if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
1357
  sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
 
1359
  else:
1360
  sk_kv = arc.keys[:, :ns].to(device=q.device, dtype=q.dtype)
1361
  sv_t = arc.values[:, :ns].to(device=q.device, dtype=q.dtype).transpose(0, 1)
1362
+ sk_t = _reposition_rope_sinks(sk_kv, ref_pos, n_blk=top.shape[1], block=block).transpose(0, 1)
1363
+ pre_k_list.append(sk_t)
1364
+ pre_v_list.append(sv_t)
1365
+ if n_sb_valid > 0:
1366
+ smk = _reposition_rope_landmarks(arc.smean_k[:, :n_sb_valid].to(dtype=q.dtype), block, ref_pos).transpose(0, 1)
1367
+ smv = arc.smean_v[:, :n_sb_valid].to(dtype=q.dtype).transpose(0, 1)
1368
+ pre_k_list.append(smk)
1369
+ pre_v_list.append(smv)
1370
+ pre_k_list.append(rk.transpose(0, 1))
1371
+ pre_v_list.append(rv.transpose(0, 1))
1372
+ pre_k = pre_k_list[0].contiguous() if len(pre_k_list) == 1 else torch.cat(pre_k_list, dim=0).contiguous()
1373
+ pre_v = pre_v_list[0].contiguous() if len(pre_v_list) == 1 else torch.cat(pre_v_list, dim=0).contiguous()
1374
  arc.cached_ns = ns
1375
  arc.cached_pre_k = pre_k
1376
  arc.cached_pre_v = pre_v
 
1389
  return torch.lerp(out_w.float(), out_p.float(), w).to(q.dtype)
1390
  elif arc.gpu_k is not None and lo >= arc.gpu_lo and hi <= arc.gpu_hi:
1391
  wk = arc.gpu_k[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
1392
+ wv = arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).transpose(0, 1) if False else arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
1393
  else:
1394
  wk = arc.keys[:, lo:hi].to(device=q.device).to(dtype=q.dtype) # [KV, Wn, D]
1395
  wv = arc.values[:, lo:hi].to(device=q.device).to(dtype=q.dtype)
 
1397
  need_mask = not (use_lse or unmasked)
1398
  parts_k, parts_v, parts_allow = [], [], []
1399
  ref_pos = lo
1400
+ n_blk_cur = top.shape[1] if top is not None else 0
1401
  if ns > 0:
1402
  if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
1403
  sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
 
1405
  else:
1406
  sk_kv = arc.keys[:, :ns].to(device=q.device).to(dtype=q.dtype)
1407
  parts_v.append(arc.values[:, :ns].to(device=q.device).to(dtype=q.dtype))
1408
+ parts_k.append(_reposition_rope_sinks(sk_kv, ref_pos, n_blk=n_blk_cur, block=block))
1409
  if need_mask:
1410
  parts_allow.append(torch.ones(t, ns, dtype=torch.bool, device=q.device))
1411
+ if n_sb_valid > 0:
1412
+ smk = _reposition_rope_landmarks(arc.smean_k[:, :n_sb_valid].to(dtype=q.dtype), block, ref_pos)
1413
+ smv = arc.smean_v[:, :n_sb_valid].to(dtype=q.dtype)
1414
+ parts_k.append(smk)
1415
+ parts_v.append(smv)
1416
+ if need_mask:
1417
+ parts_allow.append(torch.ones(t, n_sb_valid, dtype=torch.bool, device=q.device))
1418
  if top is not None:
1419
  if arc.last_top is top and arc.last_rk is not None:
1420
  rk_raw = arc.last_rk.to(dtype=q.dtype)
 
1490
  return out[0].reshape(h, t, d).transpose(0, 1).to(q.dtype)
1491
 
1492
 
1493
+ @dataclass
1494
+ class _SyntheticInputBatch:
1495
+ """Zero-hook fallback batch descriptor synthesized directly from ForwardContext / positions when
1496
+ upstream vLLM runner internals change."""
1497
+ num_tokens: int
1498
+ req_ids: list[str]
1499
+ query_start_loc_np: list[int]
1500
+
1501
+
1502
+ def _fallback_batch_from_forward_context(positions: torch.Tensor, q: torch.Tensor) -> _SyntheticInputBatch | None:
1503
+ """Synthesize an InputBatch-compatible object from vLLM ForwardContext or position resets when runner hooks
1504
+ are unavailable."""
1505
+ pos = positions[0] if positions.dim() == 2 else positions
1506
+ t = int(min(pos.numel(), q.shape[0]))
1507
+ if t <= 0:
1508
+ return None
1509
+ try:
1510
+ from vllm.forward_context import get_forward_context
1511
+ ctx = get_forward_context()
1512
+ attn_meta = getattr(ctx, "attn_metadata", None)
1513
+ if isinstance(attn_meta, dict) and attn_meta:
1514
+ attn_meta = next(iter(attn_meta.values()))
1515
+ qsl = getattr(attn_meta, "query_start_loc", None)
1516
+ if qsl is not None and isinstance(qsl, torch.Tensor) and qsl.numel() >= 2:
1517
+ qsl_list = [int(x) for x in qsl.detach().cpu().tolist() if int(x) <= t]
1518
+ if len(qsl_list) >= 2 and qsl_list[-1] > 0:
1519
+ n_req = len(qsl_list) - 1
1520
+ return _SyntheticInputBatch(
1521
+ num_tokens=qsl_list[-1],
1522
+ req_ids=[f"pb27_ctx_{i}" for i in range(n_req)],
1523
+ query_start_loc_np=qsl_list,
1524
+ )
1525
+ except Exception:
1526
+ pass
1527
+ return _SyntheticInputBatch(num_tokens=t, req_ids=["pb27_default_req"], query_start_loc_np=[0, t])
1528
+
1529
+
1530
  def install_runner_hooks() -> None:
1531
+ """Expose the runner's current InputBatch and free archives of finished requests across both vLLM V1
1532
+ (`vllm.v1.worker.gpu_model_runner`) and modular V1/V2 (`vllm.v1.worker.gpu.model_runner`) layouts."""
1533
+ try:
1534
+ from vllm.v1.worker.gpu.model_runner import GPUModelRunner
1535
+ from vllm.v1.worker.gpu.states import RequestState
1536
 
1537
+ if getattr(GPUModelRunner, "_pb27_hooked", False):
1538
+ return
1539
+ orig_prepare = GPUModelRunner.prepare_inputs
1540
+ orig_add = RequestState.add_request
1541
+ orig_remove = RequestState.remove_request
1542
+
1543
+ def prepare_inputs(self, *args, **kwargs):
1544
+ ib = orig_prepare(self, *args, **kwargs)
1545
+ BATCH["ib"] = ib
1546
+ BATCH["pos_list"] = None
1547
+ BATCH["req_bounds"] = None
1548
+ return ib
1549
+
1550
+ def add_request(self, req_id, prompt_len, *args, **kwargs):
1551
+ PROMPT_LENS[req_id] = int(prompt_len)
1552
+ all_token_ids = kwargs.get("all_token_ids", args[0] if len(args) > 0 else None)
1553
+ num_computed = kwargs.get("num_computed_tokens", args[1] if len(args) > 1 else 0)
1554
+ if all_token_ids is not None and num_computed:
1555
+ adopt_for_request(req_id, all_token_ids, int(num_computed))
1556
+ if int(prompt_len) > 262144 and not ADOPTED.get(req_id, ({},))[0] and len(RETIRED) >= max(1, RETAIN):
1557
+ while RETIRED:
1558
+ _free(RETIRED.pop(0)["archives"])
1559
+ return orig_add(self, req_id, prompt_len, *args, **kwargs)
1560
+
1561
+ def remove_request(self, req_id, *args, **kwargs):
1562
+ tokens = None
1563
+ if RETAIN > 0 and req_id in self.req_id_to_index:
1564
+ idx = self.req_id_to_index[req_id]
1565
+ try:
1566
+ n = int(self.total_len.gpu[idx])
1567
+ tokens = self.all_token_ids.gpu[idx, :n].clone()
1568
+ except Exception as exc:
1569
+ log.warning("pb27 memory: cannot read token ids for %s (%s); not retaining", req_id, exc)
1570
+ drop_request(req_id, tokens)
1571
+ return orig_remove(self, req_id, *args, **kwargs)
1572
+
1573
+ GPUModelRunner.prepare_inputs = prepare_inputs
1574
+ RequestState.add_request = add_request
1575
+ RequestState.remove_request = remove_request
1576
+ GPUModelRunner._pb27_hooked = True
1577
+ log.info("pb27 memory tier: runner hooks installed (V2 layout, pid %d), config %s", os.getpid(), MEMORY)
1578
  return
1579
+ except ImportError:
1580
+ pass
1581
+
1582
+ from vllm.v1.worker.gpu_model_runner import GPUModelRunner as V1GPUModelRunner
1583
+
1584
+ if getattr(V1GPUModelRunner, "_pb27_hooked", False):
1585
+ return
1586
+ orig_update = getattr(V1GPUModelRunner, "_update_states", None)
1587
+ if orig_update is not None:
1588
+ def _update_states(self, scheduler_output, *args, **kwargs):
1589
+ for req_id in getattr(scheduler_output, "finished_req_ids", ()):
1590
+ drop_request(req_id, None)
1591
+ for new_req in getattr(scheduler_output, "scheduled_new_reqs", ()):
1592
+ r_id = getattr(new_req, "req_id", None)
1593
+ p_ids = getattr(new_req, "prompt_token_ids", None)
1594
+ if r_id is not None and p_ids is not None:
1595
+ PROMPT_LENS[r_id] = len(p_ids)
1596
+ res = orig_update(self, scheduler_output, *args, **kwargs)
1597
+ BATCH["ib"] = getattr(self, "input_batch", None)
1598
+ BATCH["pos_list"] = None
1599
+ BATCH["req_bounds"] = None
1600
+ return res
1601
+
1602
+ V1GPUModelRunner._update_states = _update_states
1603
+ V1GPUModelRunner._pb27_hooked = True
1604
+ log.info("pb27 memory tier: runner hooks installed (V1 layout, pid %d), config %s", os.getpid(), MEMORY)
 
 
 
 
 
 
 
 
 
 
 
 
 
1605
 
1606
 
1607
  def _layer_enabled(layer_name: str) -> bool:
 
1646
  STATS["calls"] += 1
1647
  if DEBUG and STATS["calls"] % 500 == 1:
1648
  log.info("pb27 memory: call %d ib=%s archives=%d stats=%s", STATS["calls"], ib is not None, len(ARCHIVES), STATS)
1649
+ if ib is None or not getattr(ib, "req_ids", None):
1650
+ ib = _fallback_batch_from_forward_context(positions, q)
1651
+ if ib is None or not ib.req_ids:
1652
+ STATS["fallback"] += 1
1653
+ return attn_module(q, k, v)
1654
  layer = attn_module.layer_name
1655
  if not _layer_enabled(layer):
1656
  return attn_module(q, k, v)
 
1795
  budget = 0
1796
  rr = (p1 - p0) <= q_piece if PREFILL == "pt" else rerank
1797
  q_hist = None
1798
+ eff_stride = GLOBAL_DECODE_STRIDE if arc.is_global_agg else DECODE_STRIDE
1799
  need_q_hist = (
1800
  (QWIN or PERTOKEN)
1801
  and not is_prefill
1802
  and arc.qbuf is not None
1803
  and (
1804
  QWIN
1805
+ or eff_stride <= 1
1806
  or arc.cached_top is None
1807
+ or (arc.decode_step + 1) % eff_stride == 0
1808
  )
1809
  )
1810
  if need_q_hist:
pb27_vllm_plugin/start_pb27_studio.sh CHANGED
@@ -4,7 +4,9 @@
4
  set -euo pipefail
5
  ROOT=$HOME/pb27
6
  export PB27_MEMORY=${PB27_MEMORY-128,64,128,0,256}
7
- export PB27_MEMORY_SCALE=${PB27_MEMORY_SCALE-2048:384:4096:512:8192:1536}
 
 
8
  export PB27_MEMORY_PREFILL=${PB27_MEMORY_PREFILL:-pt}
9
  export PB27_MEMORY_SINKS=${PB27_MEMORY_SINKS:-4}
10
  export PB27_MEMORY_OFFLOAD=${PB27_MEMORY_OFFLOAD:-nvme}
@@ -22,7 +24,7 @@ export PB27_MEMORY_DECODE_RERANK=${PB27_MEMORY_DECODE_RERANK:-0}
22
  export PB27_MEMORY_SUPERBLOCK=${PB27_MEMORY_SUPERBLOCK:-64}
23
  export PB27_EAGER=1 # the archive/retrieval path is not CUDA-graph safe yet
24
  export GPU_MEM=${GPU_MEM:-0.75}
25
- export PB27_KV_CACHE_MEMORY=${PB27_KV_CACHE_MEMORY:-6G}
26
  export MAXLEN=${MAXLEN:-1056768}
27
  export MAX_BATCH_TOKENS=${MAX_BATCH_TOKENS:-8192}
28
  export QUANT=
 
4
  set -euo pipefail
5
  ROOT=$HOME/pb27
6
  export PB27_MEMORY=${PB27_MEMORY-128,64,128,0,256}
7
+ export PB27_MEMORY_SCALE=${PB27_MEMORY_SCALE-2048:384:4096:512:8192:1024}
8
+ export PB27_MEMORY_ROPE_HORIZON=${PB27_MEMORY_ROPE_HORIZON:-262144}
9
+ export PB27_MEMORY_ROPE_ARC_SPAN=${PB27_MEMORY_ROPE_ARC_SPAN:-98304}
10
  export PB27_MEMORY_PREFILL=${PB27_MEMORY_PREFILL:-pt}
11
  export PB27_MEMORY_SINKS=${PB27_MEMORY_SINKS:-4}
12
  export PB27_MEMORY_OFFLOAD=${PB27_MEMORY_OFFLOAD:-nvme}
 
24
  export PB27_MEMORY_SUPERBLOCK=${PB27_MEMORY_SUPERBLOCK:-64}
25
  export PB27_EAGER=1 # the archive/retrieval path is not CUDA-graph safe yet
26
  export GPU_MEM=${GPU_MEM:-0.75}
27
+ export PB27_KV_CACHE_MEMORY=${PB27_KV_CACHE_MEMORY:-4500M}
28
  export MAXLEN=${MAXLEN:-1056768}
29
  export MAX_BATCH_TOKENS=${MAX_BATCH_TOKENS:-8192}
30
  export QUANT=
pb27_vllm_plugin/studio_proxy.py CHANGED
@@ -70,6 +70,24 @@ class Handler(BaseHTTPRequestHandler):
70
  if "model" in body:
71
  body["model"] = ARGS.model
72
  body.pop("pseudobrain_mode", None) # old-server extra rejected by vLLM; keep tools/tool_choice for OpenCode & Studio
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  raw = json.dumps(body).encode()
74
  except ValueError:
75
  pass
 
70
  if "model" in body:
71
  body["model"] = ARGS.model
72
  body.pop("pseudobrain_mode", None) # old-server extra rejected by vLLM; keep tools/tool_choice for OpenCode & Studio
73
+ if self.path.startswith("/v1/chat/completions"):
74
+ ctk = body.setdefault("chat_template_kwargs", {})
75
+ if isinstance(ctk, dict):
76
+ ctk.setdefault("preserve_thinking", False)
77
+ if body.get("tools"):
78
+ ctk.setdefault("reasoning_effort", "medium")
79
+ msgs = body.get("messages")
80
+ if isinstance(msgs, list) and len(msgs) > 2:
81
+ for m in msgs[:-2]:
82
+ if isinstance(m, dict) and m.get("role") == "assistant":
83
+ m.pop("reasoning_content", None)
84
+ m.pop("reasoning", None)
85
+ if len(msgs) > 8:
86
+ for m in msgs[:-8]:
87
+ if isinstance(m, dict) and m.get("role") == "tool":
88
+ c = m.get("content")
89
+ if isinstance(c, str) and len(c) > 6000:
90
+ m["content"] = c[:2500] + "\n...[older tool output compacted]...\n" + c[-2500:]
91
  raw = json.dumps(body).encode()
92
  except ValueError:
93
  pass
pb27_vllm_plugin/switch_pb27_model.sh ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Switch the standing Spark OpenCode server (:8000 / :18080) back and forth between:
3
+ # before | pb27c -> ~/pb27/hf/pb27c_nvfp4 (28.86B pre-spatial PB-27C)
4
+ # after | spatial | pb27s -> ~/pb27/hf/pb27s_nvfp4 (30.71B PB-27S with +1.856B spatial_mlp)
5
+ # pb27b -> ~/pb27/hf/pb27b_nvfp4 (28.86B PB-27B rollback)
6
+ # status -> show active server checkpoint and saved model directories
7
+ set -euo pipefail
8
+
9
+ ROOT=$HOME/pb27
10
+ TARGET=${1:-status}
11
+
12
+ show_status() {
13
+ echo "=== Active Server (:18080) ==="
14
+ curl -s -m 3 http://127.0.0.1:18080/health || echo "(server not responding on :18080)"
15
+ echo ""
16
+ echo "=== Saved Checkpoints in $ROOT/hf ==="
17
+ for tag in pb27c pb27c_nvfp4 pb27s pb27s_nvfp4 pb27b pb27b_nvfp4 nexc nexc_nvfp4; do
18
+ if [ -d "$ROOT/hf/$tag" ]; then
19
+ echo " [READY] $ROOT/hf/$tag"
20
+ else
21
+ echo " [PENDING] $ROOT/hf/$tag"
22
+ fi
23
+ done
24
+ }
25
+
26
+ case "$TARGET" in
27
+ status|list)
28
+ show_status
29
+ exit 0
30
+ ;;
31
+ before|pb27c)
32
+ CHECKPOINT="pb27c"
33
+ MODEL_DIR="$ROOT/hf/pb27c_nvfp4"
34
+ SHA=$(sha256sum "$MODEL_DIR/pb27-gdn-00001-of-00001.safetensors" | cut -c1-64)
35
+ ;;
36
+ after|spatial|pb27s)
37
+ CHECKPOINT="pb27s"
38
+ MODEL_DIR="$ROOT/hf/pb27s_nvfp4"
39
+ if [ ! -f "$MODEL_DIR/pb27-spatial-00001-of-00001.safetensors" ]; then
40
+ echo "Error: $MODEL_DIR/pb27-spatial-00001-of-00001.safetensors is not ready yet (Colab G4 training still in progress)." >&2
41
+ exit 1
42
+ fi
43
+ SHA=$(sha256sum "$MODEL_DIR/pb27-spatial-00001-of-00001.safetensors" | cut -c1-64)
44
+ ;;
45
+ pb27b)
46
+ CHECKPOINT="pb27b"
47
+ MODEL_DIR="$ROOT/hf/pb27b_nvfp4"
48
+ SHA=$(sha256sum "$MODEL_DIR/pb27-gdn-00001-of-00001.safetensors" | cut -c1-64)
49
+ ;;
50
+ *)
51
+ echo "Usage: bash $0 {status|before|pb27c|after|spatial|pb27s|pb27b}" >&2
52
+ exit 1
53
+ ;;
54
+ esac
55
+
56
+ if [ ! -d "$MODEL_DIR" ]; then
57
+ echo "Error: model directory $MODEL_DIR does not exist." >&2
58
+ exit 1
59
+ fi
60
+
61
+ echo "Switching standing server to $CHECKPOINT ($MODEL_DIR, sha=${SHA:0:12}...)..."
62
+ docker rm -f pb27-vllm >/dev/null 2>&1 || true
63
+
64
+ PB27_MODEL_DIR="$MODEL_DIR" \
65
+ PB27_CHECKPOINT="$CHECKPOINT" \
66
+ PB27_SHA="$SHA" \
67
+ bash "$ROOT/start_pb27_studio.sh"
68
+
69
+ echo "Waiting for vLLM + DFlash2 warmup on :18080..."
70
+ for i in $(seq 1 90); do
71
+ if curl -s -m 3 http://127.0.0.1:18080/health | grep -q '"ready": true'; then
72
+ echo "Switch complete!"
73
+ curl -s -m 3 http://127.0.0.1:18080/health
74
+ echo ""
75
+ exit 0
76
+ fi
77
+ sleep 2
78
+ done
79
+
80
+ echo "Server is still warming up; check logs with: tail -f $ROOT/vllm_pb27_dflash.log"