pastapaul commited on
Commit
40cb94b
·
verified ·
1 Parent(s): dff7c56

Correction 2026-07-01: retract non-apples-to-apples MTP-on-vs-off throughput comparisons, keep verified quality and footprint claims

Browse files
Files changed (1) hide show
  1. README.md +50 -52
README.md CHANGED
@@ -23,35 +23,56 @@ An **INT4 weight-only (W4A16) quantization of GLM-5.2** that preserves the BF16
23
  layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
24
  with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).
25
 
26
- **Built for Hopper (H200).** It matches FP8 quality on **half the GPUs** (4×H200 vs 8) and is the **fastest
27
- H200-servable 4-bit GLM-5.2 in the interactive/agentic regime** — because it ships a working MTP draft head
28
- and because NVFP4 has no FP4 tensor cores on Hopper (see below).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ## Why this model
31
 
32
  - **Half the footprint, FP8 quality.** ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on
33
  **4×H200 instead of 8** — freeing half the fleet, or two replicas per node — and eval matches the FP8
34
  baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
35
- - **Fastest 4-bit GLM-5.2 on Hopper for interactive/agentic workloads.** At concurrency 1 it does **132 tok/s
36
- — +79% vs NVFP4, +69% vs AWQ-INT4, +48% vs FP8** — from MTP speculative decoding, which neither popular
37
- 4-bit competitor ships.
38
- - **Why it beats NVFP4 on Hopper specifically.** NVFP4 is a Blackwell-native FP4 format; FP4 tensor cores
39
- exist only on Blackwell (sm_100/sm_103). On an H200 NVFP4 loads and runs but **falls back to no FP4
40
- acceleration**, so it has no speed edge here while also shipping no MTP. On Hopper, this model wins; we make
41
- **no claim about Blackwell** (NVFP4's native throughput was not measured there).
42
- - **Honest trade-off.** The MTP win is at low/medium concurrency; at full saturation (c32) the no-MTP quants
43
- edge ~13–15% ahead. Both directions are shown below.
44
-
45
- ### Throughput at a glance (8×H200, vLLM bench, output tok/s, same harness)
46
-
47
- | concurrency | **This (W4A16+MTP)** | nvidia NVFP4 | cyankiwi AWQ-INT4 | FP8 baseline |
48
- |---|---|---|---|---|
49
- | **1** (interactive) | **132** | 74 | 78 | 89 |
50
- | **8** (mid) | **466** | 410 | 465 | 354 |
51
- | **32** (saturated) | 825 | 944 | 960 | 953 |
52
-
53
- Leads at c1/c8 (where latency matters for chat and agents); the simpler no-MTP quants pull ahead only once the
54
- batch is fully saturated at c32. Full eval and methodology below.
55
 
56
  ## Purpose
57
 
@@ -81,6 +102,8 @@ capability comes from the base GLM-5.2.
81
 
82
  Measured against `zai-org/GLM-5.2-FP8` under an identical setup (generative tasks via chat-completions with a
83
  16,384-token generation budget for the reasoning CoT; matched serve config with `--reasoning-parser`).
 
 
84
 
85
  | Task | This (W4A16+MTP) | FP8 baseline |
86
  |---|---|---|
@@ -105,32 +128,6 @@ On **4×H200** it serves **128K validated** (single-stream engine ceiling ~239K
105
  injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38% (7,848/20,765 draft
106
  tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a net speedup.
107
 
108
- **Throughput (8×H200, vLLM bench, output tok/s):**
109
-
110
- | concurrency | This | FP8 |
111
- |---|---|---|
112
- | 1 | 132 (+48%) | 89 |
113
- | 8 | 466 (+32%) | 354 |
114
- | 32 | 825 (−13%) | 953 |
115
-
116
- Faster than FP8 at low/medium concurrency (MTP speculative decoding helps most in the interactive regime) and
117
- slightly slower at full saturation — honest trade-off, both directions shown.
118
-
119
- **Throughput vs popular community 4-bit quants (8×H200, output tok/s, same harness):**
120
-
121
- | concurrency | This (W4A16+MTP) | cyankiwi AWQ-INT4 | nvidia NVFP4 |
122
- |---|---|---|---|
123
- | 1 | **132** | 78 | 74 |
124
- | 8 | **466** | 465 | 410 |
125
- | 32 | 825 | 960 | 944 |
126
-
127
- Against the most-downloaded H200-servable GLM-5.2 4-bit quants: this model leads the interactive/agentic
128
- regime — **+79% vs NVFP4 and +69% vs cyankiwi at concurrency 1**, +14% vs NVFP4 at 8 — because of MTP
129
- speculative decoding (neither competitor ships a usable MTP head). At full saturation (c32) the no-MTP quants
130
- are ~15% faster (MTP's draft/verify overhead stops paying off once the batch is full). NVFP4 is a
131
- Blackwell-native FP4 format; on the H200 it runs without FP4 tensor cores (not measured on Blackwell). This is
132
- a throughput comparison only — all are ~4-bit quants of the same base model, so quality is close across the field.
133
-
134
  ## Serving (vLLM ≥ 0.23, Hopper / H200)
135
 
136
  The asymmetric W4A16 MoE **requires expert parallelism** (`--enable-expert-parallel`); plain tensor-parallel
@@ -154,7 +151,7 @@ vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
154
  --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
155
  ```
156
 
157
- Validated on Hopper (H200). On Blackwell (sm100) the serving kernels need extra flags and are not yet
158
  recommended for this artifact.
159
 
160
  ## Method
@@ -172,13 +169,14 @@ asymmetric-MoE serving fix, and the Blackwell toolchain gaps).
172
 
173
  ## Limitations
174
 
175
- - Throughput is ~13% below FP8 at very high concurrency (c32); the win is at low/medium concurrency.
176
  - 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
177
  with MTP acceptance ~38% (vs ~46–52% on 8×H200).
178
  - Asymmetric weights require `--enable-expert-parallel` to serve correctly.
179
  - Recommended on Hopper; Blackwell serving needs additional kernel flags.
 
 
180
 
181
  ## Acknowledgements
182
 
183
  Built on [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT). Quantized with
184
- [llm-compressor](https://github.com/vllm-project/llm-compressor); served with [vLLM](https://github.com/vllm-project/vllm).
 
23
  layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
24
  with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).
25
 
26
+ **Built for Hopper (H200).** It **matches FP8 quality on half the GPUs** (405 GB → 4×H200 vs 8) and preserves
27
+ a working BF16 MTP draft head for speculative decoding. Quality parity and the footprint win are the solid,
28
+ verified claims; see the correction below on throughput comparisons.
29
+
30
+ ## ⚠️ Correction (2026-07-01)
31
+
32
+ Earlier versions of this card (and the companion blog) made **cross-configuration throughput comparisons that
33
+ were not apples-to-apples**, and we have retracted them:
34
+
35
+ - **"+79% vs NVFP4" / "+69% vs AWQ-INT4" (c1).** We ran those competitors with **MTP disabled**, then compared
36
+ against our MTP-on numbers. But [`nvidia/GLM-5.2-NVFP4`](https://huggingface.co/nvidia/GLM-5.2-NVFP4) **ships
37
+ an MTP head** (`num_nextn_predict_layers: 1`) — the claim that "neither competitor ships MTP" was false. We
38
+ also ran NVFP4 on **H200, where it has no FP4 tensor cores** (NVFP4 is Blackwell-native), understating it a
39
+ second way. (An explicit [`sant1an/GLM-5.2-NVFP4-MTP`](https://huggingface.co/sant1an/GLM-5.2-NVFP4-MTP) exists too.)
40
+ - **"+48% vs FP8" (c1).** Same asymmetry: `zai-org/GLM-5.2-FP8` **also ships an MTP head**
41
+ (`num_nextn_predict_layers: 1`), but we benchmarked it **MTP-off** (89 tok/s) against our **MTP-on** (132).
42
+ So "+48%" reflects **MTP-on vs MTP-off, not W4A16 vs FP8**. (FP8+MTP was not measured — the FP8 baseline was
43
+ itself difficult to serve on H200; see the repo's F026.)
44
+
45
+ **What still holds, unchanged:** quality parity vs FP8 (below — MTP is lossless for accuracy, so that eval is
46
+ unaffected), the **footprint win** (405 GB vs FP8's 704 GB → 4×H200 vs 8), our model's **absolute** throughput,
47
+ and that **MTP works** (accept ~46–52%). What we can't yet claim is any *comparative* "% faster" number.
48
+
49
+ **Being re-run (fairly):** each model measured **both MTP-off and MTP-on**, at matched settings — W4A16 vs FP8
50
+ vs NVFP4 — to separately isolate (a) the MTP speedup (same model, MTP-on vs MTP-off) and (b) the quantization
51
+ effect (matched MTP setting). B300/Blackwell numbers first (so NVFP4 gets its FP4 acceleration); an H200 re-run
52
+ to match this card's original hardware will follow. Corrected numbers will be posted here.
53
 
54
  ## Why this model
55
 
56
  - **Half the footprint, FP8 quality.** ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on
57
  **4×H200 instead of 8** — freeing half the fleet, or two replicas per node — and eval matches the FP8
58
  baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
59
+ - **Working MTP draft head.** The BF16 MTP/nextn layer is preserved and functional (spec-decode acceptance
60
+ ~46–52% aggregate, ~95% at draft position 0), giving a real speculative-decode speedup in the interactive
61
+ regime. The *magnitude* of that speedup vs other configs is being re-measured fairly (see correction).
62
+ - **Honest trade-off.** MTP helps at low/medium concurrency and its draft/verify overhead stops paying off at
63
+ full saturation. Absolute numbers below.
64
+
65
+ ### Throughput — our model, absolute (8×H200, vLLM bench, output tok/s)
66
+
67
+ | concurrency | **This (W4A16+MTP, MTP on)** |
68
+ |---|---|
69
+ | 1 (interactive) | **131.6** |
70
+ | 8 (mid) | **466.0** |
71
+ | 32 (saturated) | 825.2 |
72
+
73
+ > For reference, the FP8 baseline measured **89 / 354 / 953** at c1/c8/c32 — but **with MTP off**, so it is
74
+ > *not* directly comparable to the MTP-on numbers above (see correction). Matched MTP-on/off comparisons across
75
+ > W4A16 / FP8 / NVFP4 are being re-run.
 
 
 
76
 
77
  ## Purpose
78
 
 
102
 
103
  Measured against `zai-org/GLM-5.2-FP8` under an identical setup (generative tasks via chat-completions with a
104
  16,384-token generation budget for the reasoning CoT; matched serve config with `--reasoning-parser`).
105
+ Quality is measured with speculative decoding off, where it is exact — MTP changes latency, not outputs — so
106
+ this comparison is unaffected by the throughput correction above.
107
 
108
  | Task | This (W4A16+MTP) | FP8 baseline |
109
  |---|---|---|
 
128
  injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38% (7,848/20,765 draft
129
  tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a net speedup.
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
  ## Serving (vLLM ≥ 0.23, Hopper / H200)
132
 
133
  The asymmetric W4A16 MoE **requires expert parallelism** (`--enable-expert-parallel`); plain tensor-parallel
 
151
  --max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
152
  ```
153
 
154
+ Validated on Hopper (H200). On Blackwell (sm100/sm103) the serving kernels need extra flags and are not yet
155
  recommended for this artifact.
156
 
157
  ## Method
 
169
 
170
  ## Limitations
171
 
 
172
  - 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
173
  with MTP acceptance ~38% (vs ~46–52% on 8×H200).
174
  - Asymmetric weights require `--enable-expert-parallel` to serve correctly.
175
  - Recommended on Hopper; Blackwell serving needs additional kernel flags.
176
+ - **No validated comparative throughput number at this time.** The earlier "+X% vs NVFP4/AWQ/FP8" figures were
177
+ retracted (they compared MTP-on against MTP-off — see the correction); fair matched-MTP re-runs are in progress.
178
 
179
  ## Acknowledgements
180
 
181
  Built on [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT). Quantized with
182
+ [llm-compressor](https://github.com/vllm-project/llm-compressor); served with [vLLM](https://github.com/vllm-project/vllm).