KikoCis/MiniCPM5-2B-GGUF

KIKOCIS // 8-BIT GGUF // MEASURED AGAINST BF16

 BF16 ──────────────┐
  token_embd ── bf16 │   KLD 0.000616
  blk.0..41  ── q8_0 ├──▶ top-1 98.67 %
  output     ── bf16 │   PPL +0.0109
 ───────────────────┘

MiniCPM5-2B · Q8_0-XL

llama · 2B · 128K context · −11 % KLD vs plain Q8_0

FORMAT GGUF	SIZE 3.18 / 2.68 GB	ARCH llama, 42 layers	CONTEXT 131,072
QUANTS Q8_0-XL · Q8_0	VALIDATION KLD / PPL / top-1	RUNS ON llama.cpp · Ollama	LICENSE Apache-2.0

MiniCPM5-2B — Q8_0-XL + Q8_0 GGUF

8-bit GGUFs of OpenBMB's MiniCPM5-2B, each one measured against the BF16 original. Q8_0-XL keeps the token embeddings and the output head in BF16 and puts every transformer block in Q8_0: 11 % lower KLD and 26 % less perplexity drift than a plain Q8_0, for +0.5 GB. Runs in ~3–4 GB of RAM at 8K context.

This is OpenBMB's model, our quantization. OpenBMB also publishes their own GGUFs; we measured their Q8_0 alongside ours (table below) and it is equivalent to our plain Q8_0. What this repo adds is the XL variant and the numbers.

✅ Recommended files

Use case	File	Notes
Best fidelity at 8 bits	MiniCPM5-2B-Q8_0-XL.gguf	Lowest KLD and PPL drift here. Recommended.
Smallest 8-bit	MiniCPM5-2B-Q8_0.gguf	Standard Q8_0, 0.5 GB lighter.

📦 Files

Quant	Bits (blocks / embed + head)	File size
Q8_0-XL	8 / 16	3.18 GB
Q8_0	8 / 8	2.68 GB

The embedding table and output head are unusually large for a 2B model (130,560-token vocabulary × 2,048 = 1 GB in BF16 between the two), which is why keeping them in BF16 costs 0.5 GB and is where most of the remaining 8-bit error lives.

📊 Metrics — fidelity vs the BF16 reference

KLD (Kullback–Leibler divergence, nats): how far the quant's next-token distribution drifts from BF16. Top-1 = how often the quant's most likely token is the same as BF16's. PPL Δ = perplexity above BF16 (11.196).

File	Size GB	PPL	PPL Δ	KLD mean	KLD median	KLD p95	KLD p99	Top-1 match
BF16 reference	5.04	11.196	0	0	0	0	0	100 %
Q8_0-XL	3.18	11.207	+0.0109	0.000616	0.000375	0.00193	0.00444	98.67 %
Q8_0	2.68	11.211	+0.0147	0.000692	0.000451	0.00209	0.00463	98.57 %
Q8_0 — OpenBMB's official file, for reference	2.68	11.211	+0.0147	0.000692	0.000450	0.00209	0.00464	98.56 %

Standard errors: ±0.000005 on mean KLD, ±0.003 on PPL Δ, ±0.05 points on top-1. The XL gap is well outside them.

_{Machine-readable: metrics/quant-summary-with-kld.json / .csv. Per-file detail in reports/.}

📈 Chart

🧮 Will it fit?

Memory ≈ file size + KV cache. MiniCPM5-2B uses grouped-query attention (2 KV heads), so long context is cheap:

Context	KV cache (F16)	Q8_0-XL total	Q8_0 total
8K	0.35 GB	~3.6 GB	~3.1 GB
32K	1.4 GB	~4.6 GB	~4.1 GB
128K (native)	5.6 GB	~8.9 GB	~8.4 GB

🚀 How to run it

# llama.cpp — the GGUF carries OpenBMB's chat template (tools + reasoning)
llama-server -m MiniCPM5-2B-Q8_0-XL.gguf -c 32768 --jinja --temp 1.0 --top-p 0.95

# Ollama — Modelfiles for 8K / 32K / 128K are included
ollama run hf.co/KikoCis/MiniCPM5-2B-GGUF:Q8_0-XL
ollama create minicpm5 -f Modelfile-32k

Sampling: temperature 1.0, top_p 0.95 — OpenBMB's own defaults (generation_config.json). Reasoning: the model thinks before answering; Ollama returns that in a separate thinking field, llama-server in reasoning_content with --jinja. Tools: native function calling through the embedded chat template — use the tools parameter of the OpenAI-compatible API.

⚠️ Good to know

• At 8 bits both files are already very close to BF16; the XL gain is real but small in absolute terms. Pick XL if you have 0.5 GB to spare, plain Q8_0 otherwise.
• No weights were changed: this is a faithful quantization of OpenBMB's release.
• We have not run agentic or task benchmarks on these files — the numbers above measure fidelity to the original model, not its capability. For capability, see OpenBMB's card.

📐 Evaluation methodology

• What: KLD, perplexity and top-1 agreement of each GGUF against the BF16 GGUF converted from the same weights.
• Data: wikitext-2 test split (wiki.test.raw), first 64 chunks of 2,048 tokens.
• Tool: llama-perplexity --kl-divergence-base (BF16 logits) then --kl-divergence per file.
• Source: openbmb/MiniCPM5-2B at revision f97400052a43d642bbc6e9975e2397e3ae6a6b52.
• Load test: both files loaded and answered correctly in llama.cpp and Ollama (chat template, reasoning split).
• Date: 2026-10-10.

🔁 Provenance & reproducibility

• Exact commands: scripts/build.sh — download at the pinned revision → BF16 + Q8_0 convert → Q8_0-XL quantize → KLD/PPL.
• Checksums: reports/artifact-sha256sums.txt — shasum -a 256 -c to verify.
• Per-file results: reports/, metrics/.

🗒️ Changelog

• 2026-10-10 — v1: Q8_0-XL and Q8_0, with KLD/PPL/top-1 vs BF16 and OpenBMB's Q8_0 as a reference row.

📚 Credit & license

Model, weights and training: © OpenBMB — openbmb/MiniCPM5-2B. Quantization, measurements and Modelfiles: KikoCis. Apache-2.0, same as upstream.