Correction 2026-07-01: retract non-apples-to-apples MTP-on-vs-off throughput comparisons, keep verified quality and footprint claims
Browse files
README.md
CHANGED
|
@@ -23,35 +23,56 @@ An **INT4 weight-only (W4A16) quantization of GLM-5.2** that preserves the BF16
|
|
| 23 |
layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
|
| 24 |
with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).
|
| 25 |
|
| 26 |
-
**Built for Hopper (H200).** It matches FP8 quality on
|
| 27 |
-
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
## Why this model
|
| 31 |
|
| 32 |
- **Half the footprint, FP8 quality.** ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on
|
| 33 |
**4×H200 instead of 8** — freeing half the fleet, or two replicas per node — and eval matches the FP8
|
| 34 |
baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
|
| 35 |
-
- **
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
- **
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
Leads at c1/c8 (where latency matters for chat and agents); the simpler no-MTP quants pull ahead only once the
|
| 54 |
-
batch is fully saturated at c32. Full eval and methodology below.
|
| 55 |
|
| 56 |
## Purpose
|
| 57 |
|
|
@@ -81,6 +102,8 @@ capability comes from the base GLM-5.2.
|
|
| 81 |
|
| 82 |
Measured against `zai-org/GLM-5.2-FP8` under an identical setup (generative tasks via chat-completions with a
|
| 83 |
16,384-token generation budget for the reasoning CoT; matched serve config with `--reasoning-parser`).
|
|
|
|
|
|
|
| 84 |
|
| 85 |
| Task | This (W4A16+MTP) | FP8 baseline |
|
| 86 |
|---|---|---|
|
|
@@ -105,32 +128,6 @@ On **4×H200** it serves **128K validated** (single-stream engine ceiling ~239K
|
|
| 105 |
injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38% (7,848/20,765 draft
|
| 106 |
tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a net speedup.
|
| 107 |
|
| 108 |
-
**Throughput (8×H200, vLLM bench, output tok/s):**
|
| 109 |
-
|
| 110 |
-
| concurrency | This | FP8 |
|
| 111 |
-
|---|---|---|
|
| 112 |
-
| 1 | 132 (+48%) | 89 |
|
| 113 |
-
| 8 | 466 (+32%) | 354 |
|
| 114 |
-
| 32 | 825 (−13%) | 953 |
|
| 115 |
-
|
| 116 |
-
Faster than FP8 at low/medium concurrency (MTP speculative decoding helps most in the interactive regime) and
|
| 117 |
-
slightly slower at full saturation — honest trade-off, both directions shown.
|
| 118 |
-
|
| 119 |
-
**Throughput vs popular community 4-bit quants (8×H200, output tok/s, same harness):**
|
| 120 |
-
|
| 121 |
-
| concurrency | This (W4A16+MTP) | cyankiwi AWQ-INT4 | nvidia NVFP4 |
|
| 122 |
-
|---|---|---|---|
|
| 123 |
-
| 1 | **132** | 78 | 74 |
|
| 124 |
-
| 8 | **466** | 465 | 410 |
|
| 125 |
-
| 32 | 825 | 960 | 944 |
|
| 126 |
-
|
| 127 |
-
Against the most-downloaded H200-servable GLM-5.2 4-bit quants: this model leads the interactive/agentic
|
| 128 |
-
regime — **+79% vs NVFP4 and +69% vs cyankiwi at concurrency 1**, +14% vs NVFP4 at 8 — because of MTP
|
| 129 |
-
speculative decoding (neither competitor ships a usable MTP head). At full saturation (c32) the no-MTP quants
|
| 130 |
-
are ~15% faster (MTP's draft/verify overhead stops paying off once the batch is full). NVFP4 is a
|
| 131 |
-
Blackwell-native FP4 format; on the H200 it runs without FP4 tensor cores (not measured on Blackwell). This is
|
| 132 |
-
a throughput comparison only — all are ~4-bit quants of the same base model, so quality is close across the field.
|
| 133 |
-
|
| 134 |
## Serving (vLLM ≥ 0.23, Hopper / H200)
|
| 135 |
|
| 136 |
The asymmetric W4A16 MoE **requires expert parallelism** (`--enable-expert-parallel`); plain tensor-parallel
|
|
@@ -154,7 +151,7 @@ vllm serve <repo> --tensor-parallel-size 4 --enable-expert-parallel \
|
|
| 154 |
--max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
|
| 155 |
```
|
| 156 |
|
| 157 |
-
Validated on Hopper (H200). On Blackwell (sm100) the serving kernels need extra flags and are not yet
|
| 158 |
recommended for this artifact.
|
| 159 |
|
| 160 |
## Method
|
|
@@ -172,13 +169,14 @@ asymmetric-MoE serving fix, and the Blackwell toolchain gaps).
|
|
| 172 |
|
| 173 |
## Limitations
|
| 174 |
|
| 175 |
-
- Throughput is ~13% below FP8 at very high concurrency (c32); the win is at low/medium concurrency.
|
| 176 |
- 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
|
| 177 |
with MTP acceptance ~38% (vs ~46–52% on 8×H200).
|
| 178 |
- Asymmetric weights require `--enable-expert-parallel` to serve correctly.
|
| 179 |
- Recommended on Hopper; Blackwell serving needs additional kernel flags.
|
|
|
|
|
|
|
| 180 |
|
| 181 |
## Acknowledgements
|
| 182 |
|
| 183 |
Built on [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT). Quantized with
|
| 184 |
-
[llm-compressor](https://github.com/vllm-project/llm-compressor); served with [vLLM](https://github.com/vllm-project/vllm).
|
|
|
|
| 23 |
layer for speculative decoding. Quantized from [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)
|
| 24 |
with [llm-compressor](https://github.com/vllm-project/llm-compressor) (GPTQ).
|
| 25 |
|
| 26 |
+
**Built for Hopper (H200).** It **matches FP8 quality on half the GPUs** (405 GB → 4×H200 vs 8) and preserves
|
| 27 |
+
a working BF16 MTP draft head for speculative decoding. Quality parity and the footprint win are the solid,
|
| 28 |
+
verified claims; see the correction below on throughput comparisons.
|
| 29 |
+
|
| 30 |
+
## ⚠️ Correction (2026-07-01)
|
| 31 |
+
|
| 32 |
+
Earlier versions of this card (and the companion blog) made **cross-configuration throughput comparisons that
|
| 33 |
+
were not apples-to-apples**, and we have retracted them:
|
| 34 |
+
|
| 35 |
+
- **"+79% vs NVFP4" / "+69% vs AWQ-INT4" (c1).** We ran those competitors with **MTP disabled**, then compared
|
| 36 |
+
against our MTP-on numbers. But [`nvidia/GLM-5.2-NVFP4`](https://huggingface.co/nvidia/GLM-5.2-NVFP4) **ships
|
| 37 |
+
an MTP head** (`num_nextn_predict_layers: 1`) — the claim that "neither competitor ships MTP" was false. We
|
| 38 |
+
also ran NVFP4 on **H200, where it has no FP4 tensor cores** (NVFP4 is Blackwell-native), understating it a
|
| 39 |
+
second way. (An explicit [`sant1an/GLM-5.2-NVFP4-MTP`](https://huggingface.co/sant1an/GLM-5.2-NVFP4-MTP) exists too.)
|
| 40 |
+
- **"+48% vs FP8" (c1).** Same asymmetry: `zai-org/GLM-5.2-FP8` **also ships an MTP head**
|
| 41 |
+
(`num_nextn_predict_layers: 1`), but we benchmarked it **MTP-off** (89 tok/s) against our **MTP-on** (132).
|
| 42 |
+
So "+48%" reflects **MTP-on vs MTP-off, not W4A16 vs FP8**. (FP8+MTP was not measured — the FP8 baseline was
|
| 43 |
+
itself difficult to serve on H200; see the repo's F026.)
|
| 44 |
+
|
| 45 |
+
**What still holds, unchanged:** quality parity vs FP8 (below — MTP is lossless for accuracy, so that eval is
|
| 46 |
+
unaffected), the **footprint win** (405 GB vs FP8's 704 GB → 4×H200 vs 8), our model's **absolute** throughput,
|
| 47 |
+
and that **MTP works** (accept ~46–52%). What we can't yet claim is any *comparative* "% faster" number.
|
| 48 |
+
|
| 49 |
+
**Being re-run (fairly):** each model measured **both MTP-off and MTP-on**, at matched settings — W4A16 vs FP8
|
| 50 |
+
vs NVFP4 — to separately isolate (a) the MTP speedup (same model, MTP-on vs MTP-off) and (b) the quantization
|
| 51 |
+
effect (matched MTP setting). B300/Blackwell numbers first (so NVFP4 gets its FP4 acceleration); an H200 re-run
|
| 52 |
+
to match this card's original hardware will follow. Corrected numbers will be posted here.
|
| 53 |
|
| 54 |
## Why this model
|
| 55 |
|
| 56 |
- **Half the footprint, FP8 quality.** ~405 GB of weights (down from ~1.49 TB BF16) serve one replica on
|
| 57 |
**4×H200 instead of 8** — freeing half the fleet, or two replicas per node — and eval matches the FP8
|
| 58 |
baseline within noise across reasoning, instruction-following, long-context, and agentic coding.
|
| 59 |
+
- **Working MTP draft head.** The BF16 MTP/nextn layer is preserved and functional (spec-decode acceptance
|
| 60 |
+
~46–52% aggregate, ~95% at draft position 0), giving a real speculative-decode speedup in the interactive
|
| 61 |
+
regime. The *magnitude* of that speedup vs other configs is being re-measured fairly (see correction).
|
| 62 |
+
- **Honest trade-off.** MTP helps at low/medium concurrency and its draft/verify overhead stops paying off at
|
| 63 |
+
full saturation. Absolute numbers below.
|
| 64 |
+
|
| 65 |
+
### Throughput — our model, absolute (8×H200, vLLM bench, output tok/s)
|
| 66 |
+
|
| 67 |
+
| concurrency | **This (W4A16+MTP, MTP on)** |
|
| 68 |
+
|---|---|
|
| 69 |
+
| 1 (interactive) | **131.6** |
|
| 70 |
+
| 8 (mid) | **466.0** |
|
| 71 |
+
| 32 (saturated) | 825.2 |
|
| 72 |
+
|
| 73 |
+
> For reference, the FP8 baseline measured **89 / 354 / 953** at c1/c8/c32 — but **with MTP off**, so it is
|
| 74 |
+
> *not* directly comparable to the MTP-on numbers above (see correction). Matched MTP-on/off comparisons across
|
| 75 |
+
> W4A16 / FP8 / NVFP4 are being re-run.
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
## Purpose
|
| 78 |
|
|
|
|
| 102 |
|
| 103 |
Measured against `zai-org/GLM-5.2-FP8` under an identical setup (generative tasks via chat-completions with a
|
| 104 |
16,384-token generation budget for the reasoning CoT; matched serve config with `--reasoning-parser`).
|
| 105 |
+
Quality is measured with speculative decoding off, where it is exact — MTP changes latency, not outputs — so
|
| 106 |
+
this comparison is unaffected by the throughput correction above.
|
| 107 |
|
| 108 |
| Task | This (W4A16+MTP) | FP8 baseline |
|
| 109 |
|---|---|---|
|
|
|
|
| 128 |
injected BF16 MTP layer is healthy. On 4×H200 (TP=4, 128K) aggregate acceptance is ~38% (7,848/20,765 draft
|
| 129 |
tokens, mean accept-length ~2.9) — mildly lower under the tighter memory split but still a net speedup.
|
| 130 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
## Serving (vLLM ≥ 0.23, Hopper / H200)
|
| 132 |
|
| 133 |
The asymmetric W4A16 MoE **requires expert parallelism** (`--enable-expert-parallel`); plain tensor-parallel
|
|
|
|
| 151 |
--max-model-len 32768 --gpu-memory-utilization 0.92 --trust-remote-code
|
| 152 |
```
|
| 153 |
|
| 154 |
+
Validated on Hopper (H200). On Blackwell (sm100/sm103) the serving kernels need extra flags and are not yet
|
| 155 |
recommended for this artifact.
|
| 156 |
|
| 157 |
## Method
|
|
|
|
| 169 |
|
| 170 |
## Limitations
|
| 171 |
|
|
|
|
| 172 |
- 1M-context serving requires all 8 H200s; 4×H200 serves up to ~128K (single-stream engine ceiling ~239K),
|
| 173 |
with MTP acceptance ~38% (vs ~46–52% on 8×H200).
|
| 174 |
- Asymmetric weights require `--enable-expert-parallel` to serve correctly.
|
| 175 |
- Recommended on Hopper; Blackwell serving needs additional kernel flags.
|
| 176 |
+
- **No validated comparative throughput number at this time.** The earlier "+X% vs NVFP4/AWQ/FP8" figures were
|
| 177 |
+
retracted (they compared MTP-on against MTP-off — see the correction); fair matched-MTP re-runs are in progress.
|
| 178 |
|
| 179 |
## Acknowledgements
|
| 180 |
|
| 181 |
Built on [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) (MIT). Quantized with
|
| 182 |
+
[llm-compressor](https://github.com/vllm-project/llm-compressor); served with [vLLM](https://github.com/vllm-project/vllm).
|