Update README.md and pb27_vllm_plugin (dense global agg, non-overlapping RoPE, resilient vLLM) for defnoteatin/PseudoBrain-Qwen3-Coder-Next-BF16
Browse files- README.md +4 -2
- pb27_vllm_plugin/pb27_vllm/__init__.py +83 -32
- pb27_vllm_plugin/pb27_vllm/memory.py +362 -104
- pb27_vllm_plugin/start_pb27_studio.sh +4 -2
- pb27_vllm_plugin/studio_proxy.py +18 -0
- pb27_vllm_plugin/switch_pb27_model.sh +80 -0
README.md
CHANGED
|
@@ -38,8 +38,10 @@ In standard Transformer serving, **model weights are paid once, while KV cache m
|
|
| 38 |
|
| 39 |
**`PseudoBrain-Qwen3-Coder-Next-BF16`** (`PB27Qwen3NextForCausalLM`) solves this via the **PseudoBrain architecture**:
|
| 40 |
1. **Hybrid Windowed-Attention + Parallel Gated DeltaNet (`GDN`):** 48 layers: 36 native Gated DeltaNet + 12 full-attention layers (`[3, 7, 11, ..., 47]`) converted into 16K SWA + Parallel Gated DeltaNet + Quest Exact Memory. Recurrent GDN state is **$O(1)$ fixed size** regardless of whether context is `4K` or `1,000,000` tokens.
|
| 41 |
-
2. **Three-Tier NVMe/Page-Cache Quest Exact Memory (`piece_attention`):** On the sparse anchor layers, queries attend over `[4 StreamingLLM Sinks + Top-k Quest Exact-Memory Pages + 16,384 Sliding Window]` via FlashAttention-2 LSE-merge (`pb27_vllm/memory.py`) with **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`)**
|
| 42 |
-
3. **
|
|
|
|
|
|
|
| 43 |
|
| 44 |
### Hardware & Precision Editions for `PseudoBrain-Qwen3-Coder-Next`
|
| 45 |
|
|
|
|
| 38 |
|
| 39 |
**`PseudoBrain-Qwen3-Coder-Next-BF16`** (`PB27Qwen3NextForCausalLM`) solves this via the **PseudoBrain architecture**:
|
| 40 |
1. **Hybrid Windowed-Attention + Parallel Gated DeltaNet (`GDN`):** 48 layers: 36 native Gated DeltaNet + 12 full-attention layers (`[3, 7, 11, ..., 47]`) converted into 16K SWA + Parallel Gated DeltaNet + Quest Exact Memory. Recurrent GDN state is **$O(1)$ fixed size** regardless of whether context is `4K` or `1,000,000` tokens.
|
| 41 |
+
2. **Three-Tier NVMe/Page-Cache Quest Exact Memory (`piece_attention`):** On the sparse anchor layers, queries attend over `[4 StreamingLLM Sinks + Super-Block Landmarks + Top-k Quest Exact-Memory Pages + 16,384 Sliding Window]` via FlashAttention-2 LSE-merge (`pb27_vllm/memory.py`) with **2-Level Hierarchical `bf16` Quest Super-Block Bounds (`PB27_MEMORY_SUPERBLOCK=64`)**.
|
| 42 |
+
3. **Piecewise-Proportional, Strictly Non-Overlapping Block RoPE (`_reposition_rope_blocks`):** Enforces a mandatory `64`-token footprint per retrieved block plus proportional historical-gap scaling ($T(b_j) = P_{ ext{start}} + j \cdot B + ext{round}(rac{G_{ ext{cum},j}}{G_{ ext{tot}}} S_{ ext{slack}})$) — preserving exact `+1` local continuity across contiguous block runs, strictly preventing phase overlap between nearby non-adjacent blocks, and maintaining **99%+ proportional macro-timeline distance fidelity** at `1M` tokens without static YaRN degradation.
|
| 43 |
+
4. **Dense Global Aggregation (`PB27_MEMORY_GLOBAL_AGG="auto"`):** Automatically detects high-participation-ratio global queries (`_detect_global_agg`) and activates **(a) Resident Super-Block Summary Landmarks (`smean_k`, `smean_v`, 1 summary token per `4,096` tokens in GPU HBM)**, **(b) Stratified Super-Block Timeline Coverage**, and **(c) Novelty-Rotating Decode Working-Set Sweeps (`visit_counts`)** so multi-step reasoning sweeps across hundreds of distinct blocks over `1M` tokens.
|
| 44 |
+
5. **True $O(1)$ Constant-Speed Decode & Version-Resilient `vLLM` Plugin:** Decouples the scaled prefill retrieval budget from the constant decode budget (`PB27_MEMORY_DECODE_STRIDE=64`, `PB27_MEMORY_DECODE_BLOCKS=64`), cutting **1M-token KV cache memory by `93.5%`** (`~48.0 GB` → **`~3.1 GB`**), with dual `vLLM` V1/V2 runner support and a zero-hook `ForwardContext` fallback.
|
| 45 |
|
| 46 |
### Hardware & Precision Editions for `PseudoBrain-Qwen3-Coder-Next`
|
| 47 |
|
pb27_vllm_plugin/pb27_vllm/__init__.py
CHANGED
|
@@ -26,54 +26,102 @@ UNIVERSAL_ARCHS = (
|
|
| 26 |
)
|
| 27 |
|
| 28 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
def register() -> None:
|
|
|
|
|
|
|
| 30 |
from vllm import ModelRegistry
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
class PB27Gemma4Config(Gemma4Config):
|
| 42 |
@staticmethod
|
| 43 |
def verify_and_update_config(vllm_config) -> None:
|
| 44 |
Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
|
| 45 |
-
Gemma4Config
|
|
|
|
| 46 |
|
| 47 |
class PB27Qwen4ExpForConditionalGenerationConfig(
|
| 48 |
Qwen3_5ForConditionalGenerationConfig
|
| 49 |
):
|
| 50 |
-
"""Qwen4ExpForConditionalGenerationConfig
|
| 51 |
-
DFlash2 speculative decoding support."""
|
| 52 |
|
| 53 |
@staticmethod
|
| 54 |
def verify_and_update_config(vllm_config) -> None:
|
| 55 |
Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
|
| 56 |
text_config = vllm_config.model_config.hf_text_config
|
| 57 |
-
if text_config
|
| 58 |
raise ValueError("Qwen4Exp requires hc_count > 1")
|
| 59 |
parallel_config = vllm_config.parallel_config
|
| 60 |
-
uses_ple_or_qsa = bool(text_config
|
| 61 |
getattr(text_config, "indexer_n_heads", None) is not None
|
| 62 |
)
|
| 63 |
if uses_ple_or_qsa and (
|
| 64 |
-
parallel_config
|
|
|
|
| 65 |
):
|
| 66 |
raise NotImplementedError(
|
| 67 |
"Qwen4Exp PLE/QSA does not support dual-batch overlap or microbatching"
|
| 68 |
)
|
| 69 |
-
if text_config
|
| 70 |
raise NotImplementedError(
|
| 71 |
"Qwen4Exp N-gram PLE embedding requires pipeline_parallel_size=1 "
|
| 72 |
"because non-first pipeline ranks do not receive the raw input_ids "
|
| 73 |
"it needs. Please run with PP=1."
|
| 74 |
)
|
| 75 |
multimodal_config = vllm_config.model_config.multimodal_config
|
| 76 |
-
if multimodal_config is not None and multimodal_config
|
| 77 |
_strip_qwen4_exp_mrope(vllm_config.model_config)
|
| 78 |
spec_config = vllm_config.speculative_config
|
| 79 |
if spec_config is not None:
|
|
@@ -102,28 +150,28 @@ def register() -> None:
|
|
| 102 |
)
|
| 103 |
_strip_qwen4_exp_mrope(vllm_config.model_config)
|
| 104 |
|
|
|
|
| 105 |
for arch in ARCHS:
|
| 106 |
-
if arch not in
|
| 107 |
ModelRegistry.register_model(arch, f"pb27_vllm.model:{arch}")
|
| 108 |
for arch in LLAMA_ARCHS:
|
| 109 |
-
if arch not in
|
| 110 |
ModelRegistry.register_model(arch, f"pb27_vllm.llama_model:{arch}")
|
| 111 |
for arch in QWEN4_EXP_ARCHS:
|
| 112 |
-
if arch not in
|
| 113 |
ModelRegistry.register_model(arch, f"pb27_vllm.qwen4_exp_model:{arch}")
|
| 114 |
for arch in UNIVERSAL_ARCHS:
|
| 115 |
-
if arch not in
|
| 116 |
ModelRegistry.register_model(arch, f"pb27_vllm.universal_models:{arch}")
|
| 117 |
|
| 118 |
# The verify-and-update hooks of the stock Qwen3.5 classes (mamba SSM cache
|
| 119 |
# dtype from the HF config; the CausalLM variant also drops the M-RoPE
|
| 120 |
# fields). PB-27 keeps model_type qwen3_5(_moe) / qwen3_5(_moe)_text so they
|
| 121 |
-
# apply as is; the stock MoE classes map to the same hooks
|
| 122 |
for arch in ("PB27ForConditionalGeneration", "PB27MoeForConditionalGeneration"):
|
| 123 |
MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForConditionalGenerationConfig)
|
| 124 |
for arch in ("PB27ForCausalLM", "PB27MoeForCausalLM"):
|
| 125 |
MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForCausalLMConfig)
|
| 126 |
-
# The Llama / Qwen3Next / Qwen3Moe / Qwen3 paths only need the mamba SSM cache dtype hook (no M-RoPE fields to drop).
|
| 127 |
for arch in (
|
| 128 |
*LLAMA_ARCHS,
|
| 129 |
"PB27Qwen3NextForCausalLM",
|
|
@@ -142,12 +190,17 @@ def register() -> None:
|
|
| 142 |
PB27Qwen4ExpForCausalLMConfig,
|
| 143 |
)
|
| 144 |
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
|
| 152 |
# Exact memory tier (PB27_MEMORY env): archive bookkeeping hooks on the GPU model runner.
|
| 153 |
from . import memory
|
|
@@ -157,6 +210,4 @@ def register() -> None:
|
|
| 157 |
try:
|
| 158 |
memory.install_runner_hooks()
|
| 159 |
except Exception as exc: # the API-server process has no runner; the engine/worker processes do
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
logging.getLogger(__name__).info("pb27 memory hooks not installed here: %s", exc)
|
|
|
|
| 26 |
)
|
| 27 |
|
| 28 |
|
| 29 |
+
def _fallback_strip_mrope(model_config) -> None:
|
| 30 |
+
"""Version-tolerant fallback if vLLM moves or renames `_strip_qwen4_exp_mrope`."""
|
| 31 |
+
hf_text = getattr(model_config, "hf_text_config", None)
|
| 32 |
+
for cfg in (getattr(model_config, "hf_config", None), hf_text):
|
| 33 |
+
if cfg is None:
|
| 34 |
+
continue
|
| 35 |
+
for container_name in ("rope_parameters", "rope_scaling"):
|
| 36 |
+
rope_cfg = getattr(cfg, container_name, None)
|
| 37 |
+
if isinstance(rope_cfg, dict):
|
| 38 |
+
rope_cfg.pop("mrope_section", None)
|
| 39 |
+
rope_cfg.pop("mrope_interleaved", None)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
def register() -> None:
|
| 43 |
+
import logging
|
| 44 |
+
|
| 45 |
from vllm import ModelRegistry
|
| 46 |
+
|
| 47 |
+
log = logging.getLogger(__name__)
|
| 48 |
+
|
| 49 |
+
try:
|
| 50 |
+
from vllm.config import engram as vllm_engram
|
| 51 |
+
except Exception:
|
| 52 |
+
vllm_engram = None
|
| 53 |
+
|
| 54 |
+
try:
|
| 55 |
+
from vllm.config import vllm as vllm_cfg
|
| 56 |
+
except Exception:
|
| 57 |
+
vllm_cfg = None
|
| 58 |
+
|
| 59 |
+
try:
|
| 60 |
+
from vllm.model_executor.models.config import MODELS_CONFIG_MAP
|
| 61 |
+
except Exception:
|
| 62 |
+
MODELS_CONFIG_MAP = {}
|
| 63 |
+
|
| 64 |
+
try:
|
| 65 |
+
from vllm.model_executor.models.config import (
|
| 66 |
+
Qwen3_5ForCausalLMConfig,
|
| 67 |
+
Qwen3_5ForConditionalGenerationConfig,
|
| 68 |
+
)
|
| 69 |
+
except Exception:
|
| 70 |
+
class Qwen3_5ForConditionalGenerationConfig: # type: ignore[no-redef]
|
| 71 |
+
@staticmethod
|
| 72 |
+
def verify_and_update_config(vllm_config) -> None:
|
| 73 |
+
pass
|
| 74 |
+
|
| 75 |
+
class Qwen3_5ForCausalLMConfig(Qwen3_5ForConditionalGenerationConfig): # type: ignore[no-redef]
|
| 76 |
+
pass
|
| 77 |
+
|
| 78 |
+
try:
|
| 79 |
+
from vllm.model_executor.models.config import Gemma4Config
|
| 80 |
+
except Exception:
|
| 81 |
+
Gemma4Config = Qwen3_5ForConditionalGenerationConfig
|
| 82 |
+
|
| 83 |
+
try:
|
| 84 |
+
from vllm.model_executor.models.config import _strip_qwen4_exp_mrope
|
| 85 |
+
except Exception:
|
| 86 |
+
_strip_qwen4_exp_mrope = _fallback_strip_mrope
|
| 87 |
|
| 88 |
class PB27Gemma4Config(Gemma4Config):
|
| 89 |
@staticmethod
|
| 90 |
def verify_and_update_config(vllm_config) -> None:
|
| 91 |
Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
|
| 92 |
+
if Gemma4Config is not Qwen3_5ForConditionalGenerationConfig:
|
| 93 |
+
Gemma4Config.verify_and_update_config(vllm_config)
|
| 94 |
|
| 95 |
class PB27Qwen4ExpForConditionalGenerationConfig(
|
| 96 |
Qwen3_5ForConditionalGenerationConfig
|
| 97 |
):
|
| 98 |
+
"""Qwen4ExpForConditionalGenerationConfig plus DFlash2 speculative decoding support."""
|
|
|
|
| 99 |
|
| 100 |
@staticmethod
|
| 101 |
def verify_and_update_config(vllm_config) -> None:
|
| 102 |
Qwen3_5ForConditionalGenerationConfig.verify_and_update_config(vllm_config)
|
| 103 |
text_config = vllm_config.model_config.hf_text_config
|
| 104 |
+
if getattr(text_config, "hc_count", 4) <= 1:
|
| 105 |
raise ValueError("Qwen4Exp requires hc_count > 1")
|
| 106 |
parallel_config = vllm_config.parallel_config
|
| 107 |
+
uses_ple_or_qsa = bool(getattr(text_config, "ple_layer_ids", None)) or (
|
| 108 |
getattr(text_config, "indexer_n_heads", None) is not None
|
| 109 |
)
|
| 110 |
if uses_ple_or_qsa and (
|
| 111 |
+
getattr(parallel_config, "enable_dbo", False)
|
| 112 |
+
or getattr(parallel_config, "ubatch_size", 1) > 1
|
| 113 |
):
|
| 114 |
raise NotImplementedError(
|
| 115 |
"Qwen4Exp PLE/QSA does not support dual-batch overlap or microbatching"
|
| 116 |
)
|
| 117 |
+
if getattr(text_config, "ple_layer_ids", None) and parallel_config.pipeline_parallel_size > 1:
|
| 118 |
raise NotImplementedError(
|
| 119 |
"Qwen4Exp N-gram PLE embedding requires pipeline_parallel_size=1 "
|
| 120 |
"because non-first pipeline ranks do not receive the raw input_ids "
|
| 121 |
"it needs. Please run with PP=1."
|
| 122 |
)
|
| 123 |
multimodal_config = vllm_config.model_config.multimodal_config
|
| 124 |
+
if multimodal_config is not None and getattr(multimodal_config, "language_model_only", False):
|
| 125 |
_strip_qwen4_exp_mrope(vllm_config.model_config)
|
| 126 |
spec_config = vllm_config.speculative_config
|
| 127 |
if spec_config is not None:
|
|
|
|
| 150 |
)
|
| 151 |
_strip_qwen4_exp_mrope(vllm_config.model_config)
|
| 152 |
|
| 153 |
+
supported = set(ModelRegistry.get_supported_archs())
|
| 154 |
for arch in ARCHS:
|
| 155 |
+
if arch not in supported:
|
| 156 |
ModelRegistry.register_model(arch, f"pb27_vllm.model:{arch}")
|
| 157 |
for arch in LLAMA_ARCHS:
|
| 158 |
+
if arch not in supported:
|
| 159 |
ModelRegistry.register_model(arch, f"pb27_vllm.llama_model:{arch}")
|
| 160 |
for arch in QWEN4_EXP_ARCHS:
|
| 161 |
+
if arch not in supported:
|
| 162 |
ModelRegistry.register_model(arch, f"pb27_vllm.qwen4_exp_model:{arch}")
|
| 163 |
for arch in UNIVERSAL_ARCHS:
|
| 164 |
+
if arch not in supported:
|
| 165 |
ModelRegistry.register_model(arch, f"pb27_vllm.universal_models:{arch}")
|
| 166 |
|
| 167 |
# The verify-and-update hooks of the stock Qwen3.5 classes (mamba SSM cache
|
| 168 |
# dtype from the HF config; the CausalLM variant also drops the M-RoPE
|
| 169 |
# fields). PB-27 keeps model_type qwen3_5(_moe) / qwen3_5(_moe)_text so they
|
| 170 |
+
# apply as is; the stock MoE classes map to the same hooks.
|
| 171 |
for arch in ("PB27ForConditionalGeneration", "PB27MoeForConditionalGeneration"):
|
| 172 |
MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForConditionalGenerationConfig)
|
| 173 |
for arch in ("PB27ForCausalLM", "PB27MoeForCausalLM"):
|
| 174 |
MODELS_CONFIG_MAP.setdefault(arch, Qwen3_5ForCausalLMConfig)
|
|
|
|
| 175 |
for arch in (
|
| 176 |
*LLAMA_ARCHS,
|
| 177 |
"PB27Qwen3NextForCausalLM",
|
|
|
|
| 190 |
PB27Qwen4ExpForCausalLMConfig,
|
| 191 |
)
|
| 192 |
|
| 193 |
+
if vllm_engram is not None and hasattr(vllm_engram, "_NGRAM_LAYER_FIELDS"):
|
| 194 |
+
for arch in QWEN4_EXP_ARCHS:
|
| 195 |
+
vllm_engram._NGRAM_LAYER_FIELDS[arch] = "ple_layer_ids"
|
| 196 |
+
if vllm_cfg is not None and hasattr(vllm_cfg, "DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES"):
|
| 197 |
+
vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES = (
|
| 198 |
+
vllm_cfg.DEFAULT_BREAKABLE_CUDAGRAPH_ARCHITECTURES | frozenset(QWEN4_EXP_ARCHS)
|
| 199 |
+
)
|
| 200 |
+
if hasattr(vllm_cfg, "default_breakable_cudagraph_architectures") and hasattr(
|
| 201 |
+
vllm_cfg.default_breakable_cudagraph_architectures, "cache_clear"
|
| 202 |
+
):
|
| 203 |
+
vllm_cfg.default_breakable_cudagraph_architectures.cache_clear()
|
| 204 |
|
| 205 |
# Exact memory tier (PB27_MEMORY env): archive bookkeeping hooks on the GPU model runner.
|
| 206 |
from . import memory
|
|
|
|
| 210 |
try:
|
| 211 |
memory.install_runner_hooks()
|
| 212 |
except Exception as exc: # the API-server process has no runner; the engine/worker processes do
|
| 213 |
+
log.info("pb27 memory hooks not installed here: %s", exc)
|
|
|
|
|
|
pb27_vllm_plugin/pb27_vllm/memory.py
CHANGED
|
@@ -77,6 +77,12 @@ DECODE_STRIDE = int(os.environ.get("PB27_MEMORY_DECODE_STRIDE", "64") or 64)
|
|
| 77 |
DECODE_BLOCKS = int(os.environ.get("PB27_MEMORY_DECODE_BLOCKS", "64") or 64)
|
| 78 |
DECODE_RERANK = os.environ.get("PB27_MEMORY_DECODE_RERANK", "0") == "1"
|
| 79 |
SBLK = int(os.environ.get("PB27_MEMORY_SUPERBLOCK", "64") or 64)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
LAYER_STRIDE = int(os.environ.get("PB27_MEMORY_LAYER_STRIDE", "1") or 1)
|
| 81 |
GPU_ARC_CAP = int(os.environ.get("PB27_MEMORY_GPU_ARC_CAP", "262144") or 262144)
|
| 82 |
QBUF_CAP = 256
|
|
@@ -88,7 +94,7 @@ PROMPT_LENS: dict[str, int] = globals().get("PROMPT_LENS", {}) # req_i
|
|
| 88 |
ARCHIVES: dict[tuple[str, str], "Archive"] = globals().get("ARCHIVES", {}) # (req_id, layer_name) -> Archive
|
| 89 |
STATS: dict = globals().get("STATS", {"calls": 0, "fallback": 0, "retrievals": 0, "pieces": 0, "tokens_archived": 0,
|
| 90 |
"adopt_full": 0, "adopt_partial": 0, "adopt_none": 0, "hole_skips": 0,
|
| 91 |
-
"pertoken": 0, "pertoken_capped": 0})
|
| 92 |
# Prefix caching: vLLM skips computing the cached prefix of a new request, so the archive of the request that
|
| 93 |
# produced that prefix is retained after it finishes and adopted by the next request whose tokens match it.
|
| 94 |
RETAIN = int(os.environ.get("PB27_MEMORY_RETAIN", "0") or 0) # max retired archive sets kept (0 = off)
|
|
@@ -147,6 +153,12 @@ class Archive:
|
|
| 147 |
cached_cu: tuple | None = None # (t, w_len, p_len, cu_q, cu_w, cu_p) cached int32 cu_seqlens on GPU
|
| 148 |
smax: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block max
|
| 149 |
smin: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block min
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
mapped_files: list[tuple[int, mmap.mmap, int]] = field(default_factory=list)
|
| 151 |
last_evict_pos: int = 0
|
| 152 |
|
|
@@ -176,6 +188,10 @@ class Archive:
|
|
| 176 |
mapped=True,
|
| 177 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 178 |
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
mapped_files=[mf_k, mf_v])
|
| 180 |
if OFFLOAD == "nvme":
|
| 181 |
cap = min(max(cap, -(-max_hint // block) * block), gpu_cap)
|
|
@@ -187,7 +203,11 @@ class Archive:
|
|
| 187 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 188 |
mapped=False,
|
| 189 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 190 |
-
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
kv_dev = torch.device("cpu") if "cpu" in OFFLOAD else device
|
| 192 |
kv_dtype = torch.float8_e4m3fn if "fp8" in OFFLOAD else dtype
|
| 193 |
n_blk = cap // block
|
|
@@ -197,7 +217,11 @@ class Archive:
|
|
| 197 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 198 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 199 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 200 |
-
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
|
| 202 |
def grow(self, need: int, block: int) -> None:
|
| 203 |
cap = self.kmax.shape[1] * block if self.mapped else self.keys.shape[1]
|
|
@@ -236,16 +260,21 @@ class Archive:
|
|
| 236 |
buf[:, :cap] = old
|
| 237 |
setattr(self, name, buf)
|
| 238 |
new_n_blk = new_cap // block
|
| 239 |
-
for name in ("kmax", "kmin"):
|
| 240 |
-
old = getattr(self, name)
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 244 |
new_n_sb = (new_n_blk + SBLK - 1) // SBLK
|
| 245 |
-
for name in ("smax", "smin"):
|
| 246 |
old = getattr(self, name, None)
|
| 247 |
if old is not None and old.shape[1] < new_n_sb:
|
| 248 |
-
buf = torch.
|
| 249 |
buf[:, : old.shape[1]] = old
|
| 250 |
setattr(self, name, buf)
|
| 251 |
|
|
@@ -264,11 +293,19 @@ class Archive:
|
|
| 264 |
n_full = full_sb_end - sb0
|
| 265 |
self.smax[:, sb0:full_sb_end] = self.kmax[:, b0:b1].view(kv, n_full, SBLK, d).amax(2)
|
| 266 |
self.smin[:, sb0:full_sb_end] = self.kmin[:, b0:b1].view(kv, n_full, SBLK, d).amin(2)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
for sb in range(max(sb0, full_sb_end), sb1 + 1):
|
| 268 |
b0, b1 = sb * SBLK, min(n_blk, (sb + 1) * SBLK)
|
| 269 |
if b1 > b0:
|
| 270 |
self.smax[:, sb] = self.kmax[:, b0:b1].amax(1)
|
| 271 |
self.smin[:, sb] = self.kmin[:, b0:b1].amin(1)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
|
| 273 |
def write(self, pos: torch.Tensor, k: torch.Tensor, v: torch.Tensor, block: int,
|
| 274 |
p0: int | None = None, hi: int | None = None) -> None:
|
|
@@ -315,8 +352,13 @@ class Archive:
|
|
| 315 |
s0 = ORIGIN + b_start * block - p0
|
| 316 |
s1 = ORIGIN + b_end * block - p0
|
| 317 |
k_full = kt[:, s0:s1].view(kt.shape[0], b_end - b_start, block, kt.shape[2])
|
|
|
|
| 318 |
self.kmax[:, b_start:b_end] = k_full.amax(2).to(self.kmax.dtype)
|
| 319 |
self.kmin[:, b_start:b_end] = k_full.amin(2).to(self.kmin.dtype)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 320 |
fast_bounds = True
|
| 321 |
if contig:
|
| 322 |
if self.keys.device.type == "cpu" and kt.is_cuda and self.keys.dtype == kt.dtype and n <= 8192:
|
|
@@ -362,14 +404,23 @@ class Archive:
|
|
| 362 |
return
|
| 363 |
if self.gpu_k is not None and lo >= self.gpu_lo and hi <= self.gpu_hi:
|
| 364 |
seg = self.gpu_k[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float()
|
|
|
|
| 365 |
else:
|
| 366 |
seg = self.keys[:, lo:hi].to(device=self.kmax.device).float()
|
|
|
|
| 367 |
pad = (-seg.shape[1]) % block
|
| 368 |
if pad:
|
| 369 |
seg = torch.cat((seg, seg[:, -1:].expand(-1, pad, -1)), dim=1)
|
|
|
|
|
|
|
| 370 |
blocks = seg.view(seg.shape[0], -1, block, seg.shape[2])
|
| 371 |
self.kmax[:, lo_b: lo_b + blocks.shape[1]] = blocks.amax(2).to(self.kmax.dtype)
|
| 372 |
self.kmin[:, lo_b: lo_b + blocks.shape[1]] = blocks.amin(2).to(self.kmin.dtype)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 373 |
|
| 374 |
def refresh_bounds(self, block: int) -> None:
|
| 375 |
if self.dirty_hi < self.dirty_lo:
|
|
@@ -520,9 +571,15 @@ def _get_rope_inv_freq(device: torch.device, d_rot: int) -> torch.Tensor:
|
|
| 520 |
return inv_freq
|
| 521 |
|
| 522 |
|
| 523 |
-
def _reposition_rope_sinks(
|
| 524 |
-
|
| 525 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 526 |
if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or sk.numel() == 0:
|
| 527 |
return sk
|
| 528 |
kv, ns, d = sk.shape
|
|
@@ -530,8 +587,10 @@ def _reposition_rope_sinks(sk: torch.Tensor, ref_pos: int, span: int | None = No
|
|
| 530 |
half = d_rot // 2
|
| 531 |
if half <= 0:
|
| 532 |
return sk
|
| 533 |
-
arc_span = ROPE_ARC_SPAN if span is None else span
|
| 534 |
-
|
|
|
|
|
|
|
| 535 |
if delta_val <= 0.0:
|
| 536 |
return sk
|
| 537 |
inv_freq = _get_rope_inv_freq(sk.device, d_rot)
|
|
@@ -557,12 +616,17 @@ def _reposition_rope_blocks(
|
|
| 557 |
span: int | None = None,
|
| 558 |
prio_rank: torch.Tensor | None = None,
|
| 559 |
) -> torch.Tensor:
|
| 560 |
-
"""
|
| 561 |
-
|
| 562 |
-
|
| 563 |
-
`block` tokens
|
| 564 |
-
|
| 565 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 566 |
"""
|
| 567 |
if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or rk.numel() == 0:
|
| 568 |
return rk
|
|
@@ -575,28 +639,29 @@ def _reposition_rope_blocks(
|
|
| 575 |
if half <= 0:
|
| 576 |
return rk
|
| 577 |
idx = blk_idx.to(device=rk.device, dtype=torch.long)
|
| 578 |
-
arc_span = ROPE_ARC_SPAN if span is None else span
|
| 579 |
-
if contiguous_runs and arc_span > 0 and n_blk > 1:
|
| 580 |
-
is_succ = torch.zeros_like(idx, dtype=torch.bool)
|
| 581 |
-
is_succ[:, 1:] = idx[:, 1:] == (idx[:, :-1] + 1)
|
| 582 |
-
if prio_rank is not None and prio_rank.shape == idx.shape:
|
| 583 |
-
pr = prio_rank.to(device=idx.device)
|
| 584 |
-
is_succ[:, 1:] = is_succ[:, 1:] & ((pr[:, 1:] > pr[:, :-1]) | (pr[:, :-1] < 128))
|
| 585 |
-
step = 1
|
| 586 |
-
while step < n_blk:
|
| 587 |
-
prev_idx = torch.cat((idx[:, :step], idx[:, :-step]), dim=1)
|
| 588 |
-
prev_succ = torch.cat((torch.zeros_like(is_succ[:, :step]), is_succ[:, :-step]), dim=1)
|
| 589 |
-
idx = torch.where(is_succ, prev_idx, idx)
|
| 590 |
-
is_succ = is_succ & prev_succ
|
| 591 |
-
step <<= 1
|
| 592 |
orig_pos = (ORIGIN + idx * block).to(torch.float32)
|
| 593 |
-
if
|
| 594 |
-
end_pos = float(max(0, ref_pos - 2 * block))
|
| 595 |
-
delta = (end_pos - orig_pos).clamp(min=0.0)
|
| 596 |
-
else:
|
| 597 |
scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
|
| 598 |
target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
|
| 599 |
delta = (target_pos - orig_pos).clamp(min=0.0)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 600 |
inv_freq = _get_rope_inv_freq(rk.device, d_rot)
|
| 601 |
angles = delta.unsqueeze(-1) * inv_freq
|
| 602 |
cos = angles.cos().unsqueeze(2)
|
|
@@ -612,6 +677,40 @@ def _reposition_rope_blocks(
|
|
| 612 |
return out.view(kv, total_t, d)
|
| 613 |
|
| 614 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 615 |
def _gather_blocks(t: torch.Tensor, blk_idx: torch.Tensor, block: int) -> torch.Tensor:
|
| 616 |
"""Gather contiguous blocks `blk_idx` [KV, B] starting at ORIGIN from `t` [KV, cap, D] -> [KV, B * block, D]."""
|
| 617 |
kv, cap, d = t.shape
|
|
@@ -861,18 +960,66 @@ def _second_hop_blocks(arc: Archive, blk_ord: torch.Tensor, full: int, n_valid:
|
|
| 861 |
return cand2.gather(-1, keep2)
|
| 862 |
|
| 863 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 864 |
def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit: int, budget: int = 0,
|
| 865 |
lead: int | None = None, union: int = 0,
|
| 866 |
q_hist: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor | None] | None:
|
| 867 |
-
"""Per-query Z-score + tiered + 2-hop retrieval with [b, b+1] successor expansion
|
|
|
|
| 868 |
block = MEMORY["block_size"]
|
| 869 |
full = (limit - ORIGIN) // block
|
| 870 |
if full <= 0:
|
| 871 |
return None
|
|
|
|
| 872 |
if lead == 0 and q.shape[0] <= MEMORY["chunk"]:
|
| 873 |
arc.decode_step += 1
|
| 874 |
-
if (
|
| 875 |
-
and arc.decode_step %
|
| 876 |
return arc.cached_top, None
|
| 877 |
elif lead is not None and lead < 0:
|
| 878 |
arc.decode_step = 0
|
|
@@ -880,6 +1027,8 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 880 |
arc.cached_top = arc.cached_rk = arc.cached_rv = None
|
| 881 |
arc.cached_pre_k = arc.cached_pre_v = None
|
| 882 |
arc.cached_ns = -1
|
|
|
|
|
|
|
| 883 |
arc.refresh_bounds(block)
|
| 884 |
q_step = q[:1] if (lead == 0 and q.shape[0] > 1) else q
|
| 885 |
q_eval = (torch.cat((q_hist[:-1], q_step), dim=0)
|
|
@@ -905,7 +1054,7 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 905 |
if lead == 0 and not DECODE_RERANK:
|
| 906 |
k_dec = min(DECODE_BLOCKS if DECODE_BLOCKS > 0 else (MEMORY["decode_blocks"] or MEMORY["blocks"]), n_valid)
|
| 907 |
fine_idx = None
|
| 908 |
-
if SBLK > 1 and arc.smax is not None and arc.smin is not None and full > SBLK * 16:
|
| 909 |
n_sb = (full + SBLK - 1) // SBLK
|
| 910 |
smax_t = arc.smax[:, :n_sb].to(s_dtype).transpose(-1, -2)
|
| 911 |
smin_t = arc.smin[:, :n_sb].to(s_dtype).transpose(-1, -2)
|
|
@@ -937,6 +1086,10 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 937 |
s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
|
| 938 |
z_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 939 |
z_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 940 |
ranks = _ranks(score)
|
| 941 |
r_tier = ranks.amin(1) * n_pool + _ranks(score.amax(1))
|
| 942 |
if t_ev > 8:
|
|
@@ -956,7 +1109,11 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 956 |
(-r_tier).topk(k_sel, dim=-1).indices), dim=-1).view(kv, 3 * k_sel)
|
| 957 |
keep_ord = _dedup_priority(keep_inter, k_sel)
|
| 958 |
blk_ord = fine_idx.gather(-1, keep_ord) if fine_idx is not None else keep_ord
|
| 959 |
-
if arc.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 960 |
d_prev = arc.decode_top.clamp(max=full - 1)
|
| 961 |
k_d = min(d_prev.shape[1], blk_ord.shape[1])
|
| 962 |
d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
|
|
@@ -969,13 +1126,15 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 969 |
if arc.prompt_top is not None:
|
| 970 |
p_prev = arc.prompt_top.clamp(max=full - 1)
|
| 971 |
k_out = min(p_prev.shape[1], n_valid)
|
| 972 |
-
n_pin = max(0, k_out - dec_prio.shape[1])
|
| 973 |
merged = torch.cat((p_prev[:, :n_pin], dec_prio, p_prev[:, n_pin:]), dim=-1)
|
| 974 |
top_prio = _dedup_priority(merged, k_out)
|
| 975 |
blk, arc.last_prio_rank = top_prio.sort(-1)
|
| 976 |
else:
|
| 977 |
_, arc.last_prio_rank = dec_prio.sort(-1)
|
| 978 |
blk = dec_sorted
|
|
|
|
|
|
|
| 979 |
STATS["retrievals"] += 1
|
| 980 |
STATS["pertoken"] += 1
|
| 981 |
return blk, None
|
|
@@ -1008,6 +1167,14 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 1008 |
s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
|
| 1009 |
z_score_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 1010 |
z_score_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1011 |
ranks = _ranks(score) # [KV, T_eval, full]
|
| 1012 |
r_tier = ranks.amin(1) * full + _ranks(score.amax(1)) # [KV, full]
|
| 1013 |
if lead is not None and lead < 0 and t_ev > 36:
|
|
@@ -1082,6 +1249,10 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 1082 |
(-e_tier).topk(k, dim=-1).indices), dim=-1).view(kv, 3 * k)
|
| 1083 |
keep_ord = _dedup_priority(keep_inter, k) # [KV, k]
|
| 1084 |
blk_ord = cand.gather(-1, keep_ord) # [KV, k]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1085 |
if MEMORY["rerank"] and n_valid > 16:
|
| 1086 |
n_seed = min(8, blk_ord.shape[1])
|
| 1087 |
cur_head = blk_ord[:, :n_seed]
|
|
@@ -1097,7 +1268,7 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 1097 |
arc.decode_top = None
|
| 1098 |
merged_ord = blk_ord
|
| 1099 |
elif arc.prompt_top is not None:
|
| 1100 |
-
if arc.decode_top is not None:
|
| 1101 |
d_prev = arc.decode_top.clamp(max=full - 1)
|
| 1102 |
k_d = min(d_prev.shape[1], blk_ord.shape[1])
|
| 1103 |
d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
|
|
@@ -1116,6 +1287,8 @@ def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit:
|
|
| 1116 |
blk, arc.last_prio_rank = top_prio.sort(-1)
|
| 1117 |
if lead is not None and lead < 0:
|
| 1118 |
arc.prompt_top = top_prio
|
|
|
|
|
|
|
| 1119 |
STATS["retrievals"] += 1
|
| 1120 |
STATS["pertoken"] += 1
|
| 1121 |
return blk, None
|
|
@@ -1139,8 +1312,8 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1139 |
window: int, scale: float, own: torch.Tensor | None = None,
|
| 1140 |
q_min: int | None = None, q_max: int | None = None) -> torch.Tensor:
|
| 1141 |
"""Exact attention of piece q [T, H, D] at positions q_pos over its sliding window (from the archive) plus the
|
| 1142 |
-
retrieved blocks `top` [KV, k]
|
| 1143 |
-
Returns [T, H, D]."""
|
| 1144 |
block = MEMORY["block_size"]
|
| 1145 |
t, h, d = q.shape
|
| 1146 |
kv = h // groups
|
|
@@ -1156,6 +1329,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1156 |
fold_tail = bool(use_lse and t <= MEMORY["chunk"] and FAITHFUL and top is not None and lo > tail0)
|
| 1157 |
w_lo = tail0 if fold_tail else lo
|
| 1158 |
ns = min(SINKS, lo) # sink positions below the window
|
|
|
|
| 1159 |
if use_lse:
|
| 1160 |
if arc.gpu_k is not None and w_lo >= arc.gpu_lo and hi <= arc.gpu_hi:
|
| 1161 |
wk_t = arc.gpu_k[w_lo - arc.gpu_lo : hi - arc.gpu_lo].to(dtype=q.dtype) # [Wn, KV, D] zero-copy slice
|
|
@@ -1177,6 +1351,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1177 |
arc.cached_rv = rv
|
| 1178 |
ref_pos = lo
|
| 1179 |
rk = _reposition_rope_blocks(rk_raw, top, block, ref_pos, contiguous_runs=True, prio_rank=arc.last_prio_rank)
|
|
|
|
| 1180 |
if ns > 0:
|
| 1181 |
if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
|
| 1182 |
sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
|
|
@@ -1184,12 +1359,18 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1184 |
else:
|
| 1185 |
sk_kv = arc.keys[:, :ns].to(device=q.device, dtype=q.dtype)
|
| 1186 |
sv_t = arc.values[:, :ns].to(device=q.device, dtype=q.dtype).transpose(0, 1)
|
| 1187 |
-
sk_t = _reposition_rope_sinks(sk_kv, ref_pos).transpose(0, 1)
|
| 1188 |
-
|
| 1189 |
-
|
| 1190 |
-
|
| 1191 |
-
|
| 1192 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1193 |
arc.cached_ns = ns
|
| 1194 |
arc.cached_pre_k = pre_k
|
| 1195 |
arc.cached_pre_v = pre_v
|
|
@@ -1208,7 +1389,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1208 |
return torch.lerp(out_w.float(), out_p.float(), w).to(q.dtype)
|
| 1209 |
elif arc.gpu_k is not None and lo >= arc.gpu_lo and hi <= arc.gpu_hi:
|
| 1210 |
wk = arc.gpu_k[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
|
| 1211 |
-
wv = arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
|
| 1212 |
else:
|
| 1213 |
wk = arc.keys[:, lo:hi].to(device=q.device).to(dtype=q.dtype) # [KV, Wn, D]
|
| 1214 |
wv = arc.values[:, lo:hi].to(device=q.device).to(dtype=q.dtype)
|
|
@@ -1216,6 +1397,7 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1216 |
need_mask = not (use_lse or unmasked)
|
| 1217 |
parts_k, parts_v, parts_allow = [], [], []
|
| 1218 |
ref_pos = lo
|
|
|
|
| 1219 |
if ns > 0:
|
| 1220 |
if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
|
| 1221 |
sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
|
|
@@ -1223,9 +1405,16 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1223 |
else:
|
| 1224 |
sk_kv = arc.keys[:, :ns].to(device=q.device).to(dtype=q.dtype)
|
| 1225 |
parts_v.append(arc.values[:, :ns].to(device=q.device).to(dtype=q.dtype))
|
| 1226 |
-
parts_k.append(_reposition_rope_sinks(sk_kv, ref_pos))
|
| 1227 |
if need_mask:
|
| 1228 |
parts_allow.append(torch.ones(t, ns, dtype=torch.bool, device=q.device))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1229 |
if top is not None:
|
| 1230 |
if arc.last_top is top and arc.last_rk is not None:
|
| 1231 |
rk_raw = arc.last_rk.to(dtype=q.dtype)
|
|
@@ -1301,52 +1490,118 @@ def piece_attention(arc: Archive, q: torch.Tensor, q_pos: torch.Tensor, top: tor
|
|
| 1301 |
return out[0].reshape(h, t, d).transpose(0, 1).to(q.dtype)
|
| 1302 |
|
| 1303 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1304 |
def install_runner_hooks() -> None:
|
| 1305 |
-
"""Expose the runner's current InputBatch and free archives of finished requests
|
| 1306 |
-
|
| 1307 |
-
|
|
|
|
|
|
|
| 1308 |
|
| 1309 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1310 |
return
|
| 1311 |
-
|
| 1312 |
-
|
| 1313 |
-
|
| 1314 |
-
|
| 1315 |
-
|
| 1316 |
-
|
| 1317 |
-
|
| 1318 |
-
|
| 1319 |
-
|
| 1320 |
-
|
| 1321 |
-
|
| 1322 |
-
|
| 1323 |
-
|
| 1324 |
-
|
| 1325 |
-
|
| 1326 |
-
|
| 1327 |
-
|
| 1328 |
-
|
| 1329 |
-
|
| 1330 |
-
|
| 1331 |
-
|
| 1332 |
-
|
| 1333 |
-
|
| 1334 |
-
|
| 1335 |
-
|
| 1336 |
-
|
| 1337 |
-
try: # prompt + accepted output tokens, maintained on the GPU by the runner's post-update step
|
| 1338 |
-
n = int(self.total_len.gpu[idx])
|
| 1339 |
-
tokens = self.all_token_ids.gpu[idx, :n].clone()
|
| 1340 |
-
except Exception as exc: # internals differ: fall back to no retention
|
| 1341 |
-
log.warning("pb27 memory: cannot read token ids for %s (%s); not retaining", req_id, exc)
|
| 1342 |
-
drop_request(req_id, tokens)
|
| 1343 |
-
return orig_remove(self, req_id, *args, **kwargs)
|
| 1344 |
-
|
| 1345 |
-
GPUModelRunner.prepare_inputs = prepare_inputs
|
| 1346 |
-
RequestState.add_request = add_request
|
| 1347 |
-
RequestState.remove_request = remove_request
|
| 1348 |
-
GPUModelRunner._pb27_hooked = True
|
| 1349 |
-
log.info("pb27 memory tier: runner hooks installed (pid %d), config %s", os.getpid(), MEMORY)
|
| 1350 |
|
| 1351 |
|
| 1352 |
def _layer_enabled(layer_name: str) -> bool:
|
|
@@ -1391,9 +1646,11 @@ def windowed_attention_with_memory(attn_module, q: torch.Tensor, k: torch.Tensor
|
|
| 1391 |
STATS["calls"] += 1
|
| 1392 |
if DEBUG and STATS["calls"] % 500 == 1:
|
| 1393 |
log.info("pb27 memory: call %d ib=%s archives=%d stats=%s", STATS["calls"], ib is not None, len(ARCHIVES), STATS)
|
| 1394 |
-
if ib is None or not ib
|
| 1395 |
-
|
| 1396 |
-
|
|
|
|
|
|
|
| 1397 |
layer = attn_module.layer_name
|
| 1398 |
if not _layer_enabled(layer):
|
| 1399 |
return attn_module(q, k, v)
|
|
@@ -1538,15 +1795,16 @@ def windowed_attention_with_memory(attn_module, q: torch.Tensor, k: torch.Tensor
|
|
| 1538 |
budget = 0
|
| 1539 |
rr = (p1 - p0) <= q_piece if PREFILL == "pt" else rerank
|
| 1540 |
q_hist = None
|
|
|
|
| 1541 |
need_q_hist = (
|
| 1542 |
(QWIN or PERTOKEN)
|
| 1543 |
and not is_prefill
|
| 1544 |
and arc.qbuf is not None
|
| 1545 |
and (
|
| 1546 |
QWIN
|
| 1547 |
-
or
|
| 1548 |
or arc.cached_top is None
|
| 1549 |
-
or (arc.decode_step + 1) %
|
| 1550 |
)
|
| 1551 |
)
|
| 1552 |
if need_q_hist:
|
|
|
|
| 77 |
DECODE_BLOCKS = int(os.environ.get("PB27_MEMORY_DECODE_BLOCKS", "64") or 64)
|
| 78 |
DECODE_RERANK = os.environ.get("PB27_MEMORY_DECODE_RERANK", "0") == "1"
|
| 79 |
SBLK = int(os.environ.get("PB27_MEMORY_SUPERBLOCK", "64") or 64)
|
| 80 |
+
# Dense Global Aggregation ("auto", "1", or "0"): detects high-entropy global queries across the archive and activates
|
| 81 |
+
# (1) stratified super-block timeline coverage, (2) novelty-rotating decode working sets across decode steps, and
|
| 82 |
+
# (3) resident super-block summary KV landmarks (`smean_k` / `smean_v`, 1 summary token per 4,096 tokens).
|
| 83 |
+
GLOBAL_AGG = os.environ.get("PB27_MEMORY_GLOBAL_AGG", "auto").strip().lower()
|
| 84 |
+
GLOBAL_DECODE_STRIDE = int(os.environ.get("PB27_MEMORY_GLOBAL_DECODE_STRIDE", "16") or 16)
|
| 85 |
+
NOVELTY_ALPHA = float(os.environ.get("PB27_MEMORY_NOVELTY_ALPHA", "0.75") or 0.75)
|
| 86 |
LAYER_STRIDE = int(os.environ.get("PB27_MEMORY_LAYER_STRIDE", "1") or 1)
|
| 87 |
GPU_ARC_CAP = int(os.environ.get("PB27_MEMORY_GPU_ARC_CAP", "262144") or 262144)
|
| 88 |
QBUF_CAP = 256
|
|
|
|
| 94 |
ARCHIVES: dict[tuple[str, str], "Archive"] = globals().get("ARCHIVES", {}) # (req_id, layer_name) -> Archive
|
| 95 |
STATS: dict = globals().get("STATS", {"calls": 0, "fallback": 0, "retrievals": 0, "pieces": 0, "tokens_archived": 0,
|
| 96 |
"adopt_full": 0, "adopt_partial": 0, "adopt_none": 0, "hole_skips": 0,
|
| 97 |
+
"pertoken": 0, "pertoken_capped": 0, "global_agg": 0})
|
| 98 |
# Prefix caching: vLLM skips computing the cached prefix of a new request, so the archive of the request that
|
| 99 |
# produced that prefix is retained after it finishes and adopted by the next request whose tokens match it.
|
| 100 |
RETAIN = int(os.environ.get("PB27_MEMORY_RETAIN", "0") or 0) # max retired archive sets kept (0 = off)
|
|
|
|
| 153 |
cached_cu: tuple | None = None # (t, w_len, p_len, cu_q, cu_w, cu_p) cached int32 cu_seqlens on GPU
|
| 154 |
smax: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block max
|
| 155 |
smin: torch.Tensor | None = None # [KV, (cap // block + SBLK - 1) // SBLK, D] coarse bf16 super-block min
|
| 156 |
+
bmean_k: torch.Tensor | None = None # [KV, cap // block, D] per-block mean key on GPU for hierarchical summaries
|
| 157 |
+
bmean_v: torch.Tensor | None = None # [KV, cap // block, D] per-block mean value on GPU for hierarchical summaries
|
| 158 |
+
smean_k: torch.Tensor | None = None # [KV, n_sb, D] coarse super-block summary key landmarks on GPU
|
| 159 |
+
smean_v: torch.Tensor | None = None # [KV, n_sb, D] coarse super-block summary value landmarks on GPU
|
| 160 |
+
visit_counts: torch.Tensor | None = None # [KV, cap // block] int16 decode visit counts for novelty sweeping
|
| 161 |
+
is_global_agg: bool = False # True when Dense Global Aggregation mode is active for this request
|
| 162 |
mapped_files: list[tuple[int, mmap.mmap, int]] = field(default_factory=list)
|
| 163 |
last_evict_pos: int = 0
|
| 164 |
|
|
|
|
| 188 |
mapped=True,
|
| 189 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 190 |
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 191 |
+
bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 192 |
+
bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 193 |
+
smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 194 |
+
smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 195 |
mapped_files=[mf_k, mf_v])
|
| 196 |
if OFFLOAD == "nvme":
|
| 197 |
cap = min(max(cap, -(-max_hint // block) * block), gpu_cap)
|
|
|
|
| 203 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 204 |
mapped=False,
|
| 205 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 206 |
+
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 207 |
+
bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 208 |
+
bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 209 |
+
smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 210 |
+
smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device))
|
| 211 |
kv_dev = torch.device("cpu") if "cpu" in OFFLOAD else device
|
| 212 |
kv_dtype = torch.float8_e4m3fn if "fp8" in OFFLOAD else dtype
|
| 213 |
n_blk = cap // block
|
|
|
|
| 217 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 218 |
torch.empty(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 219 |
smax=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 220 |
+
smin=torch.empty(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 221 |
+
bmean_k=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 222 |
+
bmean_v=torch.zeros(kv, n_blk, d, dtype=b_dtype, device=device),
|
| 223 |
+
smean_k=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device),
|
| 224 |
+
smean_v=torch.zeros(kv, n_sb, d, dtype=b_dtype, device=device))
|
| 225 |
|
| 226 |
def grow(self, need: int, block: int) -> None:
|
| 227 |
cap = self.kmax.shape[1] * block if self.mapped else self.keys.shape[1]
|
|
|
|
| 260 |
buf[:, :cap] = old
|
| 261 |
setattr(self, name, buf)
|
| 262 |
new_n_blk = new_cap // block
|
| 263 |
+
for name in ("kmax", "kmin", "bmean_k", "bmean_v"):
|
| 264 |
+
old = getattr(self, name, None)
|
| 265 |
+
if old is not None:
|
| 266 |
+
buf = torch.zeros(kv, new_n_blk, d, dtype=old.dtype, device=old.device)
|
| 267 |
+
buf[:, : old.shape[1]] = old
|
| 268 |
+
setattr(self, name, buf)
|
| 269 |
+
if self.visit_counts is not None and self.visit_counts.shape[1] < new_n_blk:
|
| 270 |
+
vc = torch.zeros(kv, new_n_blk, dtype=self.visit_counts.dtype, device=self.visit_counts.device)
|
| 271 |
+
vc[:, : self.visit_counts.shape[1]] = self.visit_counts
|
| 272 |
+
self.visit_counts = vc
|
| 273 |
new_n_sb = (new_n_blk + SBLK - 1) // SBLK
|
| 274 |
+
for name in ("smax", "smin", "smean_k", "smean_v"):
|
| 275 |
old = getattr(self, name, None)
|
| 276 |
if old is not None and old.shape[1] < new_n_sb:
|
| 277 |
+
buf = torch.zeros(kv, new_n_sb, d, dtype=old.dtype, device=old.device)
|
| 278 |
buf[:, : old.shape[1]] = old
|
| 279 |
setattr(self, name, buf)
|
| 280 |
|
|
|
|
| 293 |
n_full = full_sb_end - sb0
|
| 294 |
self.smax[:, sb0:full_sb_end] = self.kmax[:, b0:b1].view(kv, n_full, SBLK, d).amax(2)
|
| 295 |
self.smin[:, sb0:full_sb_end] = self.kmin[:, b0:b1].view(kv, n_full, SBLK, d).amin(2)
|
| 296 |
+
if self.bmean_k is not None and self.smean_k is not None:
|
| 297 |
+
self.smean_k[:, sb0:full_sb_end] = self.bmean_k[:, b0:b1].view(kv, n_full, SBLK, d).mean(2).to(self.smean_k.dtype)
|
| 298 |
+
if self.bmean_v is not None and self.smean_v is not None:
|
| 299 |
+
self.smean_v[:, sb0:full_sb_end] = self.bmean_v[:, b0:b1].view(kv, n_full, SBLK, d).mean(2).to(self.smean_v.dtype)
|
| 300 |
for sb in range(max(sb0, full_sb_end), sb1 + 1):
|
| 301 |
b0, b1 = sb * SBLK, min(n_blk, (sb + 1) * SBLK)
|
| 302 |
if b1 > b0:
|
| 303 |
self.smax[:, sb] = self.kmax[:, b0:b1].amax(1)
|
| 304 |
self.smin[:, sb] = self.kmin[:, b0:b1].amin(1)
|
| 305 |
+
if self.bmean_k is not None and self.smean_k is not None:
|
| 306 |
+
self.smean_k[:, sb] = self.bmean_k[:, b0:b1].mean(1).to(self.smean_k.dtype)
|
| 307 |
+
if self.bmean_v is not None and self.smean_v is not None:
|
| 308 |
+
self.smean_v[:, sb] = self.bmean_v[:, b0:b1].mean(1).to(self.smean_v.dtype)
|
| 309 |
|
| 310 |
def write(self, pos: torch.Tensor, k: torch.Tensor, v: torch.Tensor, block: int,
|
| 311 |
p0: int | None = None, hi: int | None = None) -> None:
|
|
|
|
| 352 |
s0 = ORIGIN + b_start * block - p0
|
| 353 |
s1 = ORIGIN + b_end * block - p0
|
| 354 |
k_full = kt[:, s0:s1].view(kt.shape[0], b_end - b_start, block, kt.shape[2])
|
| 355 |
+
v_full = vt[:, s0:s1].view(vt.shape[0], b_end - b_start, block, vt.shape[2])
|
| 356 |
self.kmax[:, b_start:b_end] = k_full.amax(2).to(self.kmax.dtype)
|
| 357 |
self.kmin[:, b_start:b_end] = k_full.amin(2).to(self.kmin.dtype)
|
| 358 |
+
if self.bmean_k is not None:
|
| 359 |
+
self.bmean_k[:, b_start:b_end] = k_full.mean(2).to(self.bmean_k.dtype)
|
| 360 |
+
if self.bmean_v is not None:
|
| 361 |
+
self.bmean_v[:, b_start:b_end] = v_full.mean(2).to(self.bmean_v.dtype)
|
| 362 |
fast_bounds = True
|
| 363 |
if contig:
|
| 364 |
if self.keys.device.type == "cpu" and kt.is_cuda and self.keys.dtype == kt.dtype and n <= 8192:
|
|
|
|
| 404 |
return
|
| 405 |
if self.gpu_k is not None and lo >= self.gpu_lo and hi <= self.gpu_hi:
|
| 406 |
seg = self.gpu_k[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float()
|
| 407 |
+
seg_v = self.gpu_v[lo - self.gpu_lo : hi - self.gpu_lo].transpose(0, 1).float() if self.gpu_v is not None else None
|
| 408 |
else:
|
| 409 |
seg = self.keys[:, lo:hi].to(device=self.kmax.device).float()
|
| 410 |
+
seg_v = self.values[:, lo:hi].to(device=self.kmax.device).float() if self.bmean_v is not None else None
|
| 411 |
pad = (-seg.shape[1]) % block
|
| 412 |
if pad:
|
| 413 |
seg = torch.cat((seg, seg[:, -1:].expand(-1, pad, -1)), dim=1)
|
| 414 |
+
if seg_v is not None:
|
| 415 |
+
seg_v = torch.cat((seg_v, seg_v[:, -1:].expand(-1, pad, -1)), dim=1)
|
| 416 |
blocks = seg.view(seg.shape[0], -1, block, seg.shape[2])
|
| 417 |
self.kmax[:, lo_b: lo_b + blocks.shape[1]] = blocks.amax(2).to(self.kmax.dtype)
|
| 418 |
self.kmin[:, lo_b: lo_b + blocks.shape[1]] = blocks.amin(2).to(self.kmin.dtype)
|
| 419 |
+
if self.bmean_k is not None:
|
| 420 |
+
self.bmean_k[:, lo_b: lo_b + blocks.shape[1]] = blocks.mean(2).to(self.bmean_k.dtype)
|
| 421 |
+
if self.bmean_v is not None and seg_v is not None:
|
| 422 |
+
blocks_v = seg_v.view(seg_v.shape[0], -1, block, seg_v.shape[2])
|
| 423 |
+
self.bmean_v[:, lo_b: lo_b + blocks_v.shape[1]] = blocks_v.mean(2).to(self.bmean_v.dtype)
|
| 424 |
|
| 425 |
def refresh_bounds(self, block: int) -> None:
|
| 426 |
if self.dirty_hi < self.dirty_lo:
|
|
|
|
| 571 |
return inv_freq
|
| 572 |
|
| 573 |
|
| 574 |
+
def _reposition_rope_sinks(
|
| 575 |
+
sk: torch.Tensor,
|
| 576 |
+
ref_pos: int,
|
| 577 |
+
span: int | None = None,
|
| 578 |
+
n_blk: int = 0,
|
| 579 |
+
block: int = 64,
|
| 580 |
+
) -> torch.Tensor:
|
| 581 |
+
"""Reposition the sink keys `sk` [KV, ns, D] from original positions `0..ns-1` to `[base_start - ns, base_start)`,
|
| 582 |
+
strictly before the earliest repositioned archive block `[base_start, ref_pos)` when `ref_pos > ROPE_HORIZON`."""
|
| 583 |
if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or sk.numel() == 0:
|
| 584 |
return sk
|
| 585 |
kv, ns, d = sk.shape
|
|
|
|
| 587 |
half = d_rot // 2
|
| 588 |
if half <= 0:
|
| 589 |
return sk
|
| 590 |
+
arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
|
| 591 |
+
eff_span = max(arc_span, n_blk * block)
|
| 592 |
+
base_start = max(ORIGIN, ref_pos - eff_span)
|
| 593 |
+
delta_val = float(max(0, base_start - ns))
|
| 594 |
if delta_val <= 0.0:
|
| 595 |
return sk
|
| 596 |
inv_freq = _get_rope_inv_freq(sk.device, d_rot)
|
|
|
|
| 616 |
span: int | None = None,
|
| 617 |
prio_rank: torch.Tensor | None = None,
|
| 618 |
) -> torch.Tensor:
|
| 619 |
+
"""Piecewise-Proportional, Strictly Non-Overlapping Block RoPE into `[ref_pos - eff_span, ref_pos)`.
|
| 620 |
+
|
| 621 |
+
Guarantees three simultaneous invariants when `ref_pos > ROPE_HORIZON`:
|
| 622 |
+
1. Exact Intra-Block & Contiguous-Run Continuity: All `block` tokens inside block `b` rotate by the same shift
|
| 623 |
+
`delta_b` (preserving exact `+1` local steps), and contiguous blocks `b_j = b_{j-1} + 1` have inter-block
|
| 624 |
+
historical gap `g_mid = 0`, receiving the exact same `delta` across the block boundary.
|
| 625 |
+
2. Strict Chronological Non-Overlap: Every block `j` is allocated a mandatory `block`-token footprint plus its
|
| 626 |
+
proportional share of the remaining RoPE slack, guaranteeing `T(b_j) >= T(b_{j-1}) + block` so nearby
|
| 627 |
+
non-adjacent blocks (`b_j = b_{j-1} + 2`) never invert or overlap in rotary phase space.
|
| 628 |
+
3. Proportional Macro-Timeline Preservation: True historical gaps `g_j = (b_j - b_{j-1} - 1) * block` scale
|
| 629 |
+
proportionally across the `ROPE_ARC_SPAN` horizon instead of flattening relative timeline distances.
|
| 630 |
"""
|
| 631 |
if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or rk.numel() == 0:
|
| 632 |
return rk
|
|
|
|
| 639 |
if half <= 0:
|
| 640 |
return rk
|
| 641 |
idx = blk_idx.to(device=rk.device, dtype=torch.long)
|
| 642 |
+
arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 643 |
orig_pos = (ORIGIN + idx * block).to(torch.float32)
|
| 644 |
+
if not contiguous_runs:
|
|
|
|
|
|
|
|
|
|
| 645 |
scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
|
| 646 |
target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
|
| 647 |
delta = (target_pos - orig_pos).clamp(min=0.0)
|
| 648 |
+
else:
|
| 649 |
+
eff_span = max(arc_span, n_blk * block)
|
| 650 |
+
base_start = float(max(ORIGIN, ref_pos - eff_span))
|
| 651 |
+
slack = float(max(0.0, (ref_pos - base_start) - n_blk * block))
|
| 652 |
+
g_pre = (orig_pos[:, :1] - float(ORIGIN)).clamp(min=0.0)
|
| 653 |
+
if n_blk > 1:
|
| 654 |
+
g_mid = ((idx[:, 1:] - idx[:, :-1] - 1).clamp(min=0) * block).to(torch.float32)
|
| 655 |
+
g_cum = torch.cat((g_pre, g_pre + torch.cumsum(g_mid, dim=1)), dim=1)
|
| 656 |
+
g_mid_sum = g_mid.sum(dim=1, keepdim=True)
|
| 657 |
+
else:
|
| 658 |
+
g_cum = g_pre
|
| 659 |
+
g_mid_sum = torch.zeros_like(g_pre)
|
| 660 |
+
g_post = (float(ref_pos) - (orig_pos[:, -1:] + float(block))).clamp(min=0.0)
|
| 661 |
+
g_tot = (g_pre + g_mid_sum + g_post).clamp(min=1.0)
|
| 662 |
+
j_steps = (torch.arange(n_blk, device=rk.device, dtype=torch.float32) * float(block)).unsqueeze(0)
|
| 663 |
+
target_pos = base_start + j_steps + torch.round(g_cum * (slack / g_tot))
|
| 664 |
+
delta = (target_pos - orig_pos).clamp(min=0.0)
|
| 665 |
inv_freq = _get_rope_inv_freq(rk.device, d_rot)
|
| 666 |
angles = delta.unsqueeze(-1) * inv_freq
|
| 667 |
cos = angles.cos().unsqueeze(2)
|
|
|
|
| 677 |
return out.view(kv, total_t, d)
|
| 678 |
|
| 679 |
|
| 680 |
+
def _reposition_rope_landmarks(
|
| 681 |
+
smk: torch.Tensor,
|
| 682 |
+
block: int,
|
| 683 |
+
ref_pos: int,
|
| 684 |
+
span: int | None = None,
|
| 685 |
+
) -> torch.Tensor:
|
| 686 |
+
"""Reposition super-block centroid landmark keys `smk` [KV, n_sb, D] proportionally into `[ref_pos - arc_span, ref_pos)`."""
|
| 687 |
+
if ROPE_HORIZON <= 0 or ref_pos <= ROPE_HORIZON or smk.numel() == 0:
|
| 688 |
+
return smk
|
| 689 |
+
kv, n_sb, d = smk.shape
|
| 690 |
+
d_rot = d // 4
|
| 691 |
+
half = d_rot // 2
|
| 692 |
+
if half <= 0:
|
| 693 |
+
return smk
|
| 694 |
+
arc_span = ROPE_ARC_SPAN if (span is None or span <= 0) else span
|
| 695 |
+
sb_stride = float(SBLK * block)
|
| 696 |
+
orig_pos = float(ORIGIN) + (torch.arange(n_sb, device=smk.device, dtype=torch.float32) + 0.5) * sb_stride
|
| 697 |
+
scale = 1.0 - min(1.0, float(arc_span) / float(max(1, ref_pos)))
|
| 698 |
+
target_pos = orig_pos + torch.round((float(ref_pos) - orig_pos).clamp(min=0.0) * scale)
|
| 699 |
+
delta = (target_pos - orig_pos).clamp(min=0.0).unsqueeze(0) # [1, n_sb]
|
| 700 |
+
inv_freq = _get_rope_inv_freq(smk.device, d_rot)
|
| 701 |
+
angles = delta.unsqueeze(-1) * inv_freq # [1, n_sb, half]
|
| 702 |
+
cos = angles.cos()
|
| 703 |
+
sin = angles.sin()
|
| 704 |
+
x1 = smk[..., :half].float()
|
| 705 |
+
x2 = smk[..., half:d_rot].float()
|
| 706 |
+
r1 = (x1 * cos - x2 * sin).to(smk.dtype)
|
| 707 |
+
r2 = (x2 * cos + x1 * sin).to(smk.dtype)
|
| 708 |
+
out = smk.clone()
|
| 709 |
+
out[..., :half] = r1
|
| 710 |
+
out[..., half:d_rot] = r2
|
| 711 |
+
return out
|
| 712 |
+
|
| 713 |
+
|
| 714 |
def _gather_blocks(t: torch.Tensor, blk_idx: torch.Tensor, block: int) -> torch.Tensor:
|
| 715 |
"""Gather contiguous blocks `blk_idx` [KV, B] starting at ORIGIN from `t` [KV, cap, D] -> [KV, B * block, D]."""
|
| 716 |
kv, cap, d = t.shape
|
|
|
|
| 960 |
return cand2.gather(-1, keep2)
|
| 961 |
|
| 962 |
|
| 963 |
+
def _detect_global_agg(z_raw: torch.Tensor, full: int) -> bool:
|
| 964 |
+
"""Detect Dense Global Aggregation queries (high participation ratio across the archived timeline)."""
|
| 965 |
+
if GLOBAL_AGG in ("0", "false", "off", "no"):
|
| 966 |
+
return False
|
| 967 |
+
if GLOBAL_AGG in ("1", "true", "on", "yes", "force"):
|
| 968 |
+
return True
|
| 969 |
+
if full < 128:
|
| 970 |
+
return False
|
| 971 |
+
z_tail = z_raw[:, -min(z_raw.shape[1], 16):].amax(1) # [KV, full]
|
| 972 |
+
z_1d = (z_tail - z_tail.mean(-1, keepdim=True)) / z_tail.std(-1, keepdim=True).clamp(min=1e-6)
|
| 973 |
+
pos_mass = (z_1d - 1.0).clamp(min=0.0)
|
| 974 |
+
l1 = pos_mass.sum(dim=-1)
|
| 975 |
+
l2_sq = (pos_mass * pos_mass).sum(dim=-1).clamp(min=1e-6)
|
| 976 |
+
n_eff = float((l1 * l1 / l2_sq).mean().item())
|
| 977 |
+
z_peak = float(z_1d.amax().item())
|
| 978 |
+
return n_eff > max(64.0, 0.12 * float(full)) and z_peak < 4.5
|
| 979 |
+
|
| 980 |
+
|
| 981 |
+
def _stratified_superblock_peaks(score_1d: torch.Tensor, full: int, max_sb: int) -> torch.Tensor:
|
| 982 |
+
"""Pick the highest-scoring block inside each uniform timeline segment across `[0, full)` -> `[KV, n_seg]`."""
|
| 983 |
+
kv = score_1d.shape[0]
|
| 984 |
+
n_seg = min(max(1, max_sb), full // max(1, SBLK))
|
| 985 |
+
if n_seg <= 1:
|
| 986 |
+
return score_1d.argmax(dim=-1, keepdim=True)
|
| 987 |
+
seg_len = full // n_seg
|
| 988 |
+
usable = n_seg * seg_len
|
| 989 |
+
s_seg = score_1d[:, :usable].view(kv, n_seg, seg_len)
|
| 990 |
+
local_best = s_seg.argmax(dim=-1)
|
| 991 |
+
seg_base = (torch.arange(n_seg, device=score_1d.device, dtype=local_best.dtype) * seg_len).unsqueeze(0)
|
| 992 |
+
return seg_base + local_best
|
| 993 |
+
|
| 994 |
+
|
| 995 |
+
def _record_block_visits(arc: Archive, blk: torch.Tensor, full: int) -> None:
|
| 996 |
+
"""Increment per-block decode visit counts for novelty-penalized global aggregation sweeps."""
|
| 997 |
+
kv = blk.shape[0]
|
| 998 |
+
cap_blk = arc.kmax.shape[1]
|
| 999 |
+
if arc.visit_counts is None or arc.visit_counts.shape[1] < cap_blk:
|
| 1000 |
+
vc = torch.zeros(kv, cap_blk, dtype=torch.int16, device=blk.device)
|
| 1001 |
+
if arc.visit_counts is not None:
|
| 1002 |
+
vc[:, : arc.visit_counts.shape[1]] = arc.visit_counts
|
| 1003 |
+
arc.visit_counts = vc
|
| 1004 |
+
b_clamped = blk.clamp(min=0, max=cap_blk - 1)
|
| 1005 |
+
ones = torch.ones_like(b_clamped, dtype=torch.int16)
|
| 1006 |
+
arc.visit_counts.scatter_add_(1, b_clamped, ones)
|
| 1007 |
+
|
| 1008 |
+
|
| 1009 |
def retrieve_blocks_per_token(arc: Archive, q: torch.Tensor, groups: int, limit: int, budget: int = 0,
|
| 1010 |
lead: int | None = None, union: int = 0,
|
| 1011 |
q_hist: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor | None] | None:
|
| 1012 |
+
"""Per-query Z-score + tiered + 2-hop retrieval with [b, b+1] successor expansion, prompt-question retention,
|
| 1013 |
+
and Dense Global Aggregation support (stratified timeline coverage + novelty-rotating decode sweep)."""
|
| 1014 |
block = MEMORY["block_size"]
|
| 1015 |
full = (limit - ORIGIN) // block
|
| 1016 |
if full <= 0:
|
| 1017 |
return None
|
| 1018 |
+
eff_stride = GLOBAL_DECODE_STRIDE if arc.is_global_agg else DECODE_STRIDE
|
| 1019 |
if lead == 0 and q.shape[0] <= MEMORY["chunk"]:
|
| 1020 |
arc.decode_step += 1
|
| 1021 |
+
if (eff_stride > 1 and arc.cached_top is not None
|
| 1022 |
+
and arc.decode_step % eff_stride != 0):
|
| 1023 |
return arc.cached_top, None
|
| 1024 |
elif lead is not None and lead < 0:
|
| 1025 |
arc.decode_step = 0
|
|
|
|
| 1027 |
arc.cached_top = arc.cached_rk = arc.cached_rv = None
|
| 1028 |
arc.cached_pre_k = arc.cached_pre_v = None
|
| 1029 |
arc.cached_ns = -1
|
| 1030 |
+
if arc.visit_counts is not None:
|
| 1031 |
+
arc.visit_counts.zero_()
|
| 1032 |
arc.refresh_bounds(block)
|
| 1033 |
q_step = q[:1] if (lead == 0 and q.shape[0] > 1) else q
|
| 1034 |
q_eval = (torch.cat((q_hist[:-1], q_step), dim=0)
|
|
|
|
| 1054 |
if lead == 0 and not DECODE_RERANK:
|
| 1055 |
k_dec = min(DECODE_BLOCKS if DECODE_BLOCKS > 0 else (MEMORY["decode_blocks"] or MEMORY["blocks"]), n_valid)
|
| 1056 |
fine_idx = None
|
| 1057 |
+
if SBLK > 1 and arc.smax is not None and arc.smin is not None and full > SBLK * 16 and not arc.is_global_agg:
|
| 1058 |
n_sb = (full + SBLK - 1) // SBLK
|
| 1059 |
smax_t = arc.smax[:, :n_sb].to(s_dtype).transpose(-1, -2)
|
| 1060 |
smin_t = arc.smin[:, :n_sb].to(s_dtype).transpose(-1, -2)
|
|
|
|
| 1086 |
s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
|
| 1087 |
z_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 1088 |
z_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 1089 |
+
if arc.is_global_agg and arc.visit_counts is not None and fine_idx is None:
|
| 1090 |
+
vc_pen = NOVELTY_ALPHA * torch.log1p(arc.visit_counts[:, :full].float()).unsqueeze(1)
|
| 1091 |
+
z_raw = z_raw - vc_pen
|
| 1092 |
+
z_rel = z_rel - vc_pen
|
| 1093 |
ranks = _ranks(score)
|
| 1094 |
r_tier = ranks.amin(1) * n_pool + _ranks(score.amax(1))
|
| 1095 |
if t_ev > 8:
|
|
|
|
| 1109 |
(-r_tier).topk(k_sel, dim=-1).indices), dim=-1).view(kv, 3 * k_sel)
|
| 1110 |
keep_ord = _dedup_priority(keep_inter, k_sel)
|
| 1111 |
blk_ord = fine_idx.gather(-1, keep_ord) if fine_idx is not None else keep_ord
|
| 1112 |
+
if arc.is_global_agg and fine_idx is None and full >= SBLK * 2:
|
| 1113 |
+
strat = _stratified_superblock_peaks(z_raw.amax(1), full, max(8, k_sel // 2))
|
| 1114 |
+
n_top = max(1, k_sel // 2)
|
| 1115 |
+
blk_ord = _dedup_priority(torch.cat((blk_ord[:, :n_top], strat, blk_ord[:, n_top:]), dim=-1), k_sel)
|
| 1116 |
+
if arc.decode_top is not None and not arc.is_global_agg:
|
| 1117 |
d_prev = arc.decode_top.clamp(max=full - 1)
|
| 1118 |
k_d = min(d_prev.shape[1], blk_ord.shape[1])
|
| 1119 |
d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
|
|
|
|
| 1126 |
if arc.prompt_top is not None:
|
| 1127 |
p_prev = arc.prompt_top.clamp(max=full - 1)
|
| 1128 |
k_out = min(p_prev.shape[1], n_valid)
|
| 1129 |
+
n_pin = (k_out // 2) if arc.is_global_agg else max(0, k_out - dec_prio.shape[1])
|
| 1130 |
merged = torch.cat((p_prev[:, :n_pin], dec_prio, p_prev[:, n_pin:]), dim=-1)
|
| 1131 |
top_prio = _dedup_priority(merged, k_out)
|
| 1132 |
blk, arc.last_prio_rank = top_prio.sort(-1)
|
| 1133 |
else:
|
| 1134 |
_, arc.last_prio_rank = dec_prio.sort(-1)
|
| 1135 |
blk = dec_sorted
|
| 1136 |
+
if arc.is_global_agg:
|
| 1137 |
+
_record_block_visits(arc, dec_prio, full)
|
| 1138 |
STATS["retrievals"] += 1
|
| 1139 |
STATS["pertoken"] += 1
|
| 1140 |
return blk, None
|
|
|
|
| 1167 |
s_rel_fin = s_rel.masked_fill(score.isinf(), 0.0)
|
| 1168 |
z_score_rel = (s_rel - s_rel_fin.mean(-1, keepdim=True)) / s_rel_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 1169 |
z_score_raw = (score - s_fin.mean(-1, keepdim=True)) / s_fin.std(-1, keepdim=True).clamp(min=1e-6)
|
| 1170 |
+
if lead is not None and lead < 0:
|
| 1171 |
+
arc.is_global_agg = _detect_global_agg(z_score_raw, full)
|
| 1172 |
+
if arc.is_global_agg:
|
| 1173 |
+
STATS["global_agg"] = STATS.get("global_agg", 0) + 1
|
| 1174 |
+
elif lead == 0 and arc.is_global_agg and arc.visit_counts is not None:
|
| 1175 |
+
vc_pen = NOVELTY_ALPHA * torch.log1p(arc.visit_counts[:, :full].float()).unsqueeze(1)
|
| 1176 |
+
z_score_raw = z_score_raw - vc_pen
|
| 1177 |
+
z_score_rel = z_score_rel - vc_pen
|
| 1178 |
ranks = _ranks(score) # [KV, T_eval, full]
|
| 1179 |
r_tier = ranks.amin(1) * full + _ranks(score.amax(1)) # [KV, full]
|
| 1180 |
if lead is not None and lead < 0 and t_ev > 36:
|
|
|
|
| 1249 |
(-e_tier).topk(k, dim=-1).indices), dim=-1).view(kv, 3 * k)
|
| 1250 |
keep_ord = _dedup_priority(keep_inter, k) # [KV, k]
|
| 1251 |
blk_ord = cand.gather(-1, keep_ord) # [KV, k]
|
| 1252 |
+
if arc.is_global_agg and full >= SBLK * 2:
|
| 1253 |
+
strat = _stratified_superblock_peaks(z_score_raw.amax(1), full, max(16, k // 4))
|
| 1254 |
+
n_keep_top = max(1, (3 * k) // 4)
|
| 1255 |
+
blk_ord = _dedup_priority(torch.cat((blk_ord[:, :n_keep_top], strat, blk_ord[:, n_keep_top:]), dim=-1), k)
|
| 1256 |
if MEMORY["rerank"] and n_valid > 16:
|
| 1257 |
n_seed = min(8, blk_ord.shape[1])
|
| 1258 |
cur_head = blk_ord[:, :n_seed]
|
|
|
|
| 1268 |
arc.decode_top = None
|
| 1269 |
merged_ord = blk_ord
|
| 1270 |
elif arc.prompt_top is not None:
|
| 1271 |
+
if arc.decode_top is not None and not arc.is_global_agg:
|
| 1272 |
d_prev = arc.decode_top.clamp(max=full - 1)
|
| 1273 |
k_d = min(d_prev.shape[1], blk_ord.shape[1])
|
| 1274 |
d_inter = torch.stack((blk_ord[:, :k_d], d_prev[:, :k_d]), dim=-1).view(kv, 2 * k_d)
|
|
|
|
| 1287 |
blk, arc.last_prio_rank = top_prio.sort(-1)
|
| 1288 |
if lead is not None and lead < 0:
|
| 1289 |
arc.prompt_top = top_prio
|
| 1290 |
+
if arc.is_global_agg:
|
| 1291 |
+
_record_block_visits(arc, top_prio, full)
|
| 1292 |
STATS["retrievals"] += 1
|
| 1293 |
STATS["pertoken"] += 1
|
| 1294 |
return blk, None
|
|
|
|
| 1312 |
window: int, scale: float, own: torch.Tensor | None = None,
|
| 1313 |
q_min: int | None = None, q_max: int | None = None) -> torch.Tensor:
|
| 1314 |
"""Exact attention of piece q [T, H, D] at positions q_pos over its sliding window (from the archive) plus the
|
| 1315 |
+
retrieved blocks `top` [KV, k] (and super-block summary landmarks when `arc.is_global_agg` is active);
|
| 1316 |
+
with `own` [KV, T, k] each query sees only its own retrieved blocks. Returns [T, H, D]."""
|
| 1317 |
block = MEMORY["block_size"]
|
| 1318 |
t, h, d = q.shape
|
| 1319 |
kv = h // groups
|
|
|
|
| 1329 |
fold_tail = bool(use_lse and t <= MEMORY["chunk"] and FAITHFUL and top is not None and lo > tail0)
|
| 1330 |
w_lo = tail0 if fold_tail else lo
|
| 1331 |
ns = min(SINKS, lo) # sink positions below the window
|
| 1332 |
+
n_sb_valid = max(0, ((lo - ORIGIN) // block) // SBLK) if (arc.is_global_agg and arc.smean_k is not None and arc.smean_v is not None) else 0
|
| 1333 |
if use_lse:
|
| 1334 |
if arc.gpu_k is not None and w_lo >= arc.gpu_lo and hi <= arc.gpu_hi:
|
| 1335 |
wk_t = arc.gpu_k[w_lo - arc.gpu_lo : hi - arc.gpu_lo].to(dtype=q.dtype) # [Wn, KV, D] zero-copy slice
|
|
|
|
| 1351 |
arc.cached_rv = rv
|
| 1352 |
ref_pos = lo
|
| 1353 |
rk = _reposition_rope_blocks(rk_raw, top, block, ref_pos, contiguous_runs=True, prio_rank=arc.last_prio_rank)
|
| 1354 |
+
pre_k_list, pre_v_list = [], []
|
| 1355 |
if ns > 0:
|
| 1356 |
if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
|
| 1357 |
sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
|
|
|
|
| 1359 |
else:
|
| 1360 |
sk_kv = arc.keys[:, :ns].to(device=q.device, dtype=q.dtype)
|
| 1361 |
sv_t = arc.values[:, :ns].to(device=q.device, dtype=q.dtype).transpose(0, 1)
|
| 1362 |
+
sk_t = _reposition_rope_sinks(sk_kv, ref_pos, n_blk=top.shape[1], block=block).transpose(0, 1)
|
| 1363 |
+
pre_k_list.append(sk_t)
|
| 1364 |
+
pre_v_list.append(sv_t)
|
| 1365 |
+
if n_sb_valid > 0:
|
| 1366 |
+
smk = _reposition_rope_landmarks(arc.smean_k[:, :n_sb_valid].to(dtype=q.dtype), block, ref_pos).transpose(0, 1)
|
| 1367 |
+
smv = arc.smean_v[:, :n_sb_valid].to(dtype=q.dtype).transpose(0, 1)
|
| 1368 |
+
pre_k_list.append(smk)
|
| 1369 |
+
pre_v_list.append(smv)
|
| 1370 |
+
pre_k_list.append(rk.transpose(0, 1))
|
| 1371 |
+
pre_v_list.append(rv.transpose(0, 1))
|
| 1372 |
+
pre_k = pre_k_list[0].contiguous() if len(pre_k_list) == 1 else torch.cat(pre_k_list, dim=0).contiguous()
|
| 1373 |
+
pre_v = pre_v_list[0].contiguous() if len(pre_v_list) == 1 else torch.cat(pre_v_list, dim=0).contiguous()
|
| 1374 |
arc.cached_ns = ns
|
| 1375 |
arc.cached_pre_k = pre_k
|
| 1376 |
arc.cached_pre_v = pre_v
|
|
|
|
| 1389 |
return torch.lerp(out_w.float(), out_p.float(), w).to(q.dtype)
|
| 1390 |
elif arc.gpu_k is not None and lo >= arc.gpu_lo and hi <= arc.gpu_hi:
|
| 1391 |
wk = arc.gpu_k[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
|
| 1392 |
+
wv = arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).transpose(0, 1) if False else arc.gpu_v[lo - arc.gpu_lo : hi - arc.gpu_lo].transpose(0, 1).to(dtype=q.dtype)
|
| 1393 |
else:
|
| 1394 |
wk = arc.keys[:, lo:hi].to(device=q.device).to(dtype=q.dtype) # [KV, Wn, D]
|
| 1395 |
wv = arc.values[:, lo:hi].to(device=q.device).to(dtype=q.dtype)
|
|
|
|
| 1397 |
need_mask = not (use_lse or unmasked)
|
| 1398 |
parts_k, parts_v, parts_allow = [], [], []
|
| 1399 |
ref_pos = lo
|
| 1400 |
+
n_blk_cur = top.shape[1] if top is not None else 0
|
| 1401 |
if ns > 0:
|
| 1402 |
if arc.sink_k is not None and ns <= arc.sink_k.shape[0]:
|
| 1403 |
sk_kv = arc.sink_k[:ns].transpose(0, 1).to(dtype=q.dtype)
|
|
|
|
| 1405 |
else:
|
| 1406 |
sk_kv = arc.keys[:, :ns].to(device=q.device).to(dtype=q.dtype)
|
| 1407 |
parts_v.append(arc.values[:, :ns].to(device=q.device).to(dtype=q.dtype))
|
| 1408 |
+
parts_k.append(_reposition_rope_sinks(sk_kv, ref_pos, n_blk=n_blk_cur, block=block))
|
| 1409 |
if need_mask:
|
| 1410 |
parts_allow.append(torch.ones(t, ns, dtype=torch.bool, device=q.device))
|
| 1411 |
+
if n_sb_valid > 0:
|
| 1412 |
+
smk = _reposition_rope_landmarks(arc.smean_k[:, :n_sb_valid].to(dtype=q.dtype), block, ref_pos)
|
| 1413 |
+
smv = arc.smean_v[:, :n_sb_valid].to(dtype=q.dtype)
|
| 1414 |
+
parts_k.append(smk)
|
| 1415 |
+
parts_v.append(smv)
|
| 1416 |
+
if need_mask:
|
| 1417 |
+
parts_allow.append(torch.ones(t, n_sb_valid, dtype=torch.bool, device=q.device))
|
| 1418 |
if top is not None:
|
| 1419 |
if arc.last_top is top and arc.last_rk is not None:
|
| 1420 |
rk_raw = arc.last_rk.to(dtype=q.dtype)
|
|
|
|
| 1490 |
return out[0].reshape(h, t, d).transpose(0, 1).to(q.dtype)
|
| 1491 |
|
| 1492 |
|
| 1493 |
+
@dataclass
|
| 1494 |
+
class _SyntheticInputBatch:
|
| 1495 |
+
"""Zero-hook fallback batch descriptor synthesized directly from ForwardContext / positions when
|
| 1496 |
+
upstream vLLM runner internals change."""
|
| 1497 |
+
num_tokens: int
|
| 1498 |
+
req_ids: list[str]
|
| 1499 |
+
query_start_loc_np: list[int]
|
| 1500 |
+
|
| 1501 |
+
|
| 1502 |
+
def _fallback_batch_from_forward_context(positions: torch.Tensor, q: torch.Tensor) -> _SyntheticInputBatch | None:
|
| 1503 |
+
"""Synthesize an InputBatch-compatible object from vLLM ForwardContext or position resets when runner hooks
|
| 1504 |
+
are unavailable."""
|
| 1505 |
+
pos = positions[0] if positions.dim() == 2 else positions
|
| 1506 |
+
t = int(min(pos.numel(), q.shape[0]))
|
| 1507 |
+
if t <= 0:
|
| 1508 |
+
return None
|
| 1509 |
+
try:
|
| 1510 |
+
from vllm.forward_context import get_forward_context
|
| 1511 |
+
ctx = get_forward_context()
|
| 1512 |
+
attn_meta = getattr(ctx, "attn_metadata", None)
|
| 1513 |
+
if isinstance(attn_meta, dict) and attn_meta:
|
| 1514 |
+
attn_meta = next(iter(attn_meta.values()))
|
| 1515 |
+
qsl = getattr(attn_meta, "query_start_loc", None)
|
| 1516 |
+
if qsl is not None and isinstance(qsl, torch.Tensor) and qsl.numel() >= 2:
|
| 1517 |
+
qsl_list = [int(x) for x in qsl.detach().cpu().tolist() if int(x) <= t]
|
| 1518 |
+
if len(qsl_list) >= 2 and qsl_list[-1] > 0:
|
| 1519 |
+
n_req = len(qsl_list) - 1
|
| 1520 |
+
return _SyntheticInputBatch(
|
| 1521 |
+
num_tokens=qsl_list[-1],
|
| 1522 |
+
req_ids=[f"pb27_ctx_{i}" for i in range(n_req)],
|
| 1523 |
+
query_start_loc_np=qsl_list,
|
| 1524 |
+
)
|
| 1525 |
+
except Exception:
|
| 1526 |
+
pass
|
| 1527 |
+
return _SyntheticInputBatch(num_tokens=t, req_ids=["pb27_default_req"], query_start_loc_np=[0, t])
|
| 1528 |
+
|
| 1529 |
+
|
| 1530 |
def install_runner_hooks() -> None:
|
| 1531 |
+
"""Expose the runner's current InputBatch and free archives of finished requests across both vLLM V1
|
| 1532 |
+
(`vllm.v1.worker.gpu_model_runner`) and modular V1/V2 (`vllm.v1.worker.gpu.model_runner`) layouts."""
|
| 1533 |
+
try:
|
| 1534 |
+
from vllm.v1.worker.gpu.model_runner import GPUModelRunner
|
| 1535 |
+
from vllm.v1.worker.gpu.states import RequestState
|
| 1536 |
|
| 1537 |
+
if getattr(GPUModelRunner, "_pb27_hooked", False):
|
| 1538 |
+
return
|
| 1539 |
+
orig_prepare = GPUModelRunner.prepare_inputs
|
| 1540 |
+
orig_add = RequestState.add_request
|
| 1541 |
+
orig_remove = RequestState.remove_request
|
| 1542 |
+
|
| 1543 |
+
def prepare_inputs(self, *args, **kwargs):
|
| 1544 |
+
ib = orig_prepare(self, *args, **kwargs)
|
| 1545 |
+
BATCH["ib"] = ib
|
| 1546 |
+
BATCH["pos_list"] = None
|
| 1547 |
+
BATCH["req_bounds"] = None
|
| 1548 |
+
return ib
|
| 1549 |
+
|
| 1550 |
+
def add_request(self, req_id, prompt_len, *args, **kwargs):
|
| 1551 |
+
PROMPT_LENS[req_id] = int(prompt_len)
|
| 1552 |
+
all_token_ids = kwargs.get("all_token_ids", args[0] if len(args) > 0 else None)
|
| 1553 |
+
num_computed = kwargs.get("num_computed_tokens", args[1] if len(args) > 1 else 0)
|
| 1554 |
+
if all_token_ids is not None and num_computed:
|
| 1555 |
+
adopt_for_request(req_id, all_token_ids, int(num_computed))
|
| 1556 |
+
if int(prompt_len) > 262144 and not ADOPTED.get(req_id, ({},))[0] and len(RETIRED) >= max(1, RETAIN):
|
| 1557 |
+
while RETIRED:
|
| 1558 |
+
_free(RETIRED.pop(0)["archives"])
|
| 1559 |
+
return orig_add(self, req_id, prompt_len, *args, **kwargs)
|
| 1560 |
+
|
| 1561 |
+
def remove_request(self, req_id, *args, **kwargs):
|
| 1562 |
+
tokens = None
|
| 1563 |
+
if RETAIN > 0 and req_id in self.req_id_to_index:
|
| 1564 |
+
idx = self.req_id_to_index[req_id]
|
| 1565 |
+
try:
|
| 1566 |
+
n = int(self.total_len.gpu[idx])
|
| 1567 |
+
tokens = self.all_token_ids.gpu[idx, :n].clone()
|
| 1568 |
+
except Exception as exc:
|
| 1569 |
+
log.warning("pb27 memory: cannot read token ids for %s (%s); not retaining", req_id, exc)
|
| 1570 |
+
drop_request(req_id, tokens)
|
| 1571 |
+
return orig_remove(self, req_id, *args, **kwargs)
|
| 1572 |
+
|
| 1573 |
+
GPUModelRunner.prepare_inputs = prepare_inputs
|
| 1574 |
+
RequestState.add_request = add_request
|
| 1575 |
+
RequestState.remove_request = remove_request
|
| 1576 |
+
GPUModelRunner._pb27_hooked = True
|
| 1577 |
+
log.info("pb27 memory tier: runner hooks installed (V2 layout, pid %d), config %s", os.getpid(), MEMORY)
|
| 1578 |
return
|
| 1579 |
+
except ImportError:
|
| 1580 |
+
pass
|
| 1581 |
+
|
| 1582 |
+
from vllm.v1.worker.gpu_model_runner import GPUModelRunner as V1GPUModelRunner
|
| 1583 |
+
|
| 1584 |
+
if getattr(V1GPUModelRunner, "_pb27_hooked", False):
|
| 1585 |
+
return
|
| 1586 |
+
orig_update = getattr(V1GPUModelRunner, "_update_states", None)
|
| 1587 |
+
if orig_update is not None:
|
| 1588 |
+
def _update_states(self, scheduler_output, *args, **kwargs):
|
| 1589 |
+
for req_id in getattr(scheduler_output, "finished_req_ids", ()):
|
| 1590 |
+
drop_request(req_id, None)
|
| 1591 |
+
for new_req in getattr(scheduler_output, "scheduled_new_reqs", ()):
|
| 1592 |
+
r_id = getattr(new_req, "req_id", None)
|
| 1593 |
+
p_ids = getattr(new_req, "prompt_token_ids", None)
|
| 1594 |
+
if r_id is not None and p_ids is not None:
|
| 1595 |
+
PROMPT_LENS[r_id] = len(p_ids)
|
| 1596 |
+
res = orig_update(self, scheduler_output, *args, **kwargs)
|
| 1597 |
+
BATCH["ib"] = getattr(self, "input_batch", None)
|
| 1598 |
+
BATCH["pos_list"] = None
|
| 1599 |
+
BATCH["req_bounds"] = None
|
| 1600 |
+
return res
|
| 1601 |
+
|
| 1602 |
+
V1GPUModelRunner._update_states = _update_states
|
| 1603 |
+
V1GPUModelRunner._pb27_hooked = True
|
| 1604 |
+
log.info("pb27 memory tier: runner hooks installed (V1 layout, pid %d), config %s", os.getpid(), MEMORY)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1605 |
|
| 1606 |
|
| 1607 |
def _layer_enabled(layer_name: str) -> bool:
|
|
|
|
| 1646 |
STATS["calls"] += 1
|
| 1647 |
if DEBUG and STATS["calls"] % 500 == 1:
|
| 1648 |
log.info("pb27 memory: call %d ib=%s archives=%d stats=%s", STATS["calls"], ib is not None, len(ARCHIVES), STATS)
|
| 1649 |
+
if ib is None or not getattr(ib, "req_ids", None):
|
| 1650 |
+
ib = _fallback_batch_from_forward_context(positions, q)
|
| 1651 |
+
if ib is None or not ib.req_ids:
|
| 1652 |
+
STATS["fallback"] += 1
|
| 1653 |
+
return attn_module(q, k, v)
|
| 1654 |
layer = attn_module.layer_name
|
| 1655 |
if not _layer_enabled(layer):
|
| 1656 |
return attn_module(q, k, v)
|
|
|
|
| 1795 |
budget = 0
|
| 1796 |
rr = (p1 - p0) <= q_piece if PREFILL == "pt" else rerank
|
| 1797 |
q_hist = None
|
| 1798 |
+
eff_stride = GLOBAL_DECODE_STRIDE if arc.is_global_agg else DECODE_STRIDE
|
| 1799 |
need_q_hist = (
|
| 1800 |
(QWIN or PERTOKEN)
|
| 1801 |
and not is_prefill
|
| 1802 |
and arc.qbuf is not None
|
| 1803 |
and (
|
| 1804 |
QWIN
|
| 1805 |
+
or eff_stride <= 1
|
| 1806 |
or arc.cached_top is None
|
| 1807 |
+
or (arc.decode_step + 1) % eff_stride == 0
|
| 1808 |
)
|
| 1809 |
)
|
| 1810 |
if need_q_hist:
|
pb27_vllm_plugin/start_pb27_studio.sh
CHANGED
|
@@ -4,7 +4,9 @@
|
|
| 4 |
set -euo pipefail
|
| 5 |
ROOT=$HOME/pb27
|
| 6 |
export PB27_MEMORY=${PB27_MEMORY-128,64,128,0,256}
|
| 7 |
-
export PB27_MEMORY_SCALE=${PB27_MEMORY_SCALE-2048:384:4096:512:8192:
|
|
|
|
|
|
|
| 8 |
export PB27_MEMORY_PREFILL=${PB27_MEMORY_PREFILL:-pt}
|
| 9 |
export PB27_MEMORY_SINKS=${PB27_MEMORY_SINKS:-4}
|
| 10 |
export PB27_MEMORY_OFFLOAD=${PB27_MEMORY_OFFLOAD:-nvme}
|
|
@@ -22,7 +24,7 @@ export PB27_MEMORY_DECODE_RERANK=${PB27_MEMORY_DECODE_RERANK:-0}
|
|
| 22 |
export PB27_MEMORY_SUPERBLOCK=${PB27_MEMORY_SUPERBLOCK:-64}
|
| 23 |
export PB27_EAGER=1 # the archive/retrieval path is not CUDA-graph safe yet
|
| 24 |
export GPU_MEM=${GPU_MEM:-0.75}
|
| 25 |
-
export PB27_KV_CACHE_MEMORY=${PB27_KV_CACHE_MEMORY:-
|
| 26 |
export MAXLEN=${MAXLEN:-1056768}
|
| 27 |
export MAX_BATCH_TOKENS=${MAX_BATCH_TOKENS:-8192}
|
| 28 |
export QUANT=
|
|
|
|
| 4 |
set -euo pipefail
|
| 5 |
ROOT=$HOME/pb27
|
| 6 |
export PB27_MEMORY=${PB27_MEMORY-128,64,128,0,256}
|
| 7 |
+
export PB27_MEMORY_SCALE=${PB27_MEMORY_SCALE-2048:384:4096:512:8192:1024}
|
| 8 |
+
export PB27_MEMORY_ROPE_HORIZON=${PB27_MEMORY_ROPE_HORIZON:-262144}
|
| 9 |
+
export PB27_MEMORY_ROPE_ARC_SPAN=${PB27_MEMORY_ROPE_ARC_SPAN:-98304}
|
| 10 |
export PB27_MEMORY_PREFILL=${PB27_MEMORY_PREFILL:-pt}
|
| 11 |
export PB27_MEMORY_SINKS=${PB27_MEMORY_SINKS:-4}
|
| 12 |
export PB27_MEMORY_OFFLOAD=${PB27_MEMORY_OFFLOAD:-nvme}
|
|
|
|
| 24 |
export PB27_MEMORY_SUPERBLOCK=${PB27_MEMORY_SUPERBLOCK:-64}
|
| 25 |
export PB27_EAGER=1 # the archive/retrieval path is not CUDA-graph safe yet
|
| 26 |
export GPU_MEM=${GPU_MEM:-0.75}
|
| 27 |
+
export PB27_KV_CACHE_MEMORY=${PB27_KV_CACHE_MEMORY:-4500M}
|
| 28 |
export MAXLEN=${MAXLEN:-1056768}
|
| 29 |
export MAX_BATCH_TOKENS=${MAX_BATCH_TOKENS:-8192}
|
| 30 |
export QUANT=
|
pb27_vllm_plugin/studio_proxy.py
CHANGED
|
@@ -70,6 +70,24 @@ class Handler(BaseHTTPRequestHandler):
|
|
| 70 |
if "model" in body:
|
| 71 |
body["model"] = ARGS.model
|
| 72 |
body.pop("pseudobrain_mode", None) # old-server extra rejected by vLLM; keep tools/tool_choice for OpenCode & Studio
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
raw = json.dumps(body).encode()
|
| 74 |
except ValueError:
|
| 75 |
pass
|
|
|
|
| 70 |
if "model" in body:
|
| 71 |
body["model"] = ARGS.model
|
| 72 |
body.pop("pseudobrain_mode", None) # old-server extra rejected by vLLM; keep tools/tool_choice for OpenCode & Studio
|
| 73 |
+
if self.path.startswith("/v1/chat/completions"):
|
| 74 |
+
ctk = body.setdefault("chat_template_kwargs", {})
|
| 75 |
+
if isinstance(ctk, dict):
|
| 76 |
+
ctk.setdefault("preserve_thinking", False)
|
| 77 |
+
if body.get("tools"):
|
| 78 |
+
ctk.setdefault("reasoning_effort", "medium")
|
| 79 |
+
msgs = body.get("messages")
|
| 80 |
+
if isinstance(msgs, list) and len(msgs) > 2:
|
| 81 |
+
for m in msgs[:-2]:
|
| 82 |
+
if isinstance(m, dict) and m.get("role") == "assistant":
|
| 83 |
+
m.pop("reasoning_content", None)
|
| 84 |
+
m.pop("reasoning", None)
|
| 85 |
+
if len(msgs) > 8:
|
| 86 |
+
for m in msgs[:-8]:
|
| 87 |
+
if isinstance(m, dict) and m.get("role") == "tool":
|
| 88 |
+
c = m.get("content")
|
| 89 |
+
if isinstance(c, str) and len(c) > 6000:
|
| 90 |
+
m["content"] = c[:2500] + "\n...[older tool output compacted]...\n" + c[-2500:]
|
| 91 |
raw = json.dumps(body).encode()
|
| 92 |
except ValueError:
|
| 93 |
pass
|
pb27_vllm_plugin/switch_pb27_model.sh
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Switch the standing Spark OpenCode server (:8000 / :18080) back and forth between:
|
| 3 |
+
# before | pb27c -> ~/pb27/hf/pb27c_nvfp4 (28.86B pre-spatial PB-27C)
|
| 4 |
+
# after | spatial | pb27s -> ~/pb27/hf/pb27s_nvfp4 (30.71B PB-27S with +1.856B spatial_mlp)
|
| 5 |
+
# pb27b -> ~/pb27/hf/pb27b_nvfp4 (28.86B PB-27B rollback)
|
| 6 |
+
# status -> show active server checkpoint and saved model directories
|
| 7 |
+
set -euo pipefail
|
| 8 |
+
|
| 9 |
+
ROOT=$HOME/pb27
|
| 10 |
+
TARGET=${1:-status}
|
| 11 |
+
|
| 12 |
+
show_status() {
|
| 13 |
+
echo "=== Active Server (:18080) ==="
|
| 14 |
+
curl -s -m 3 http://127.0.0.1:18080/health || echo "(server not responding on :18080)"
|
| 15 |
+
echo ""
|
| 16 |
+
echo "=== Saved Checkpoints in $ROOT/hf ==="
|
| 17 |
+
for tag in pb27c pb27c_nvfp4 pb27s pb27s_nvfp4 pb27b pb27b_nvfp4 nexc nexc_nvfp4; do
|
| 18 |
+
if [ -d "$ROOT/hf/$tag" ]; then
|
| 19 |
+
echo " [READY] $ROOT/hf/$tag"
|
| 20 |
+
else
|
| 21 |
+
echo " [PENDING] $ROOT/hf/$tag"
|
| 22 |
+
fi
|
| 23 |
+
done
|
| 24 |
+
}
|
| 25 |
+
|
| 26 |
+
case "$TARGET" in
|
| 27 |
+
status|list)
|
| 28 |
+
show_status
|
| 29 |
+
exit 0
|
| 30 |
+
;;
|
| 31 |
+
before|pb27c)
|
| 32 |
+
CHECKPOINT="pb27c"
|
| 33 |
+
MODEL_DIR="$ROOT/hf/pb27c_nvfp4"
|
| 34 |
+
SHA=$(sha256sum "$MODEL_DIR/pb27-gdn-00001-of-00001.safetensors" | cut -c1-64)
|
| 35 |
+
;;
|
| 36 |
+
after|spatial|pb27s)
|
| 37 |
+
CHECKPOINT="pb27s"
|
| 38 |
+
MODEL_DIR="$ROOT/hf/pb27s_nvfp4"
|
| 39 |
+
if [ ! -f "$MODEL_DIR/pb27-spatial-00001-of-00001.safetensors" ]; then
|
| 40 |
+
echo "Error: $MODEL_DIR/pb27-spatial-00001-of-00001.safetensors is not ready yet (Colab G4 training still in progress)." >&2
|
| 41 |
+
exit 1
|
| 42 |
+
fi
|
| 43 |
+
SHA=$(sha256sum "$MODEL_DIR/pb27-spatial-00001-of-00001.safetensors" | cut -c1-64)
|
| 44 |
+
;;
|
| 45 |
+
pb27b)
|
| 46 |
+
CHECKPOINT="pb27b"
|
| 47 |
+
MODEL_DIR="$ROOT/hf/pb27b_nvfp4"
|
| 48 |
+
SHA=$(sha256sum "$MODEL_DIR/pb27-gdn-00001-of-00001.safetensors" | cut -c1-64)
|
| 49 |
+
;;
|
| 50 |
+
*)
|
| 51 |
+
echo "Usage: bash $0 {status|before|pb27c|after|spatial|pb27s|pb27b}" >&2
|
| 52 |
+
exit 1
|
| 53 |
+
;;
|
| 54 |
+
esac
|
| 55 |
+
|
| 56 |
+
if [ ! -d "$MODEL_DIR" ]; then
|
| 57 |
+
echo "Error: model directory $MODEL_DIR does not exist." >&2
|
| 58 |
+
exit 1
|
| 59 |
+
fi
|
| 60 |
+
|
| 61 |
+
echo "Switching standing server to $CHECKPOINT ($MODEL_DIR, sha=${SHA:0:12}...)..."
|
| 62 |
+
docker rm -f pb27-vllm >/dev/null 2>&1 || true
|
| 63 |
+
|
| 64 |
+
PB27_MODEL_DIR="$MODEL_DIR" \
|
| 65 |
+
PB27_CHECKPOINT="$CHECKPOINT" \
|
| 66 |
+
PB27_SHA="$SHA" \
|
| 67 |
+
bash "$ROOT/start_pb27_studio.sh"
|
| 68 |
+
|
| 69 |
+
echo "Waiting for vLLM + DFlash2 warmup on :18080..."
|
| 70 |
+
for i in $(seq 1 90); do
|
| 71 |
+
if curl -s -m 3 http://127.0.0.1:18080/health | grep -q '"ready": true'; then
|
| 72 |
+
echo "Switch complete!"
|
| 73 |
+
curl -s -m 3 http://127.0.0.1:18080/health
|
| 74 |
+
echo ""
|
| 75 |
+
exit 0
|
| 76 |
+
fi
|
| 77 |
+
sleep 2
|
| 78 |
+
done
|
| 79 |
+
|
| 80 |
+
echo "Server is still warming up; check logs with: tail -f $ROOT/vllm_pb27_dflash.log"
|