gemma-4-A4B-98e-v7-coderx โ€” NVFP4A16 (vLLM)

NVFP4A16 quantization of ManniX-ITA/gemma-4-A4B-98e-v7-coderx-it, the code-maximal prune of Gemma 4 26B-A4B (128โ†’98 experts/layer, ~20.8B). 2 shards, ~13.4 GB โ€” fits a single 16 GB GPU with room for KV-cache.

NVFP4A16 = 4-bit NVFP4 weights with FP8 (e4m3) block scales and bf16 activations (A16). Quantized with NVIDIA TensorRT-Model-Optimizer (modelopt) main (0.45.0.dev), whose _QuantFusedExperts plugin handles Gemma 4's fused MoE experts. Every exported weight/scale tensor is finiteness-verified before upload.

Serve with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model ManniX-ITA/gemma-4-A4B-98e-v7-coderx-NVFP4A16 \
    --served-model-name v7-coderx \
    --port 8000 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 65536 \
    --max-num-batched-tokens 8192 \
    --dtype bfloat16 \
    --trust-remote-code \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

Gemma 4 gotchas (all required): --max-num-batched-tokens 8192 (the MM-encoder budget; the default 2048 < max_tokens_per_mm_item and crashes at boot), --max-model-len 65536 (templates may request up to ~49k gen tokens), and the gemma4 reasoning parser for the thinking format. NVFP4 needs a Blackwell/Hopper or Ada GPU on a recent vLLM.

Benchmarks

NVFP4A16 is a deployment format and is not separately benchmarked (cohort policy). The table is the cohort's same-host Q6_K ยท llama.cpp ยท greedy reference (temperature 0.0, top_p 1.0, top_k 0), read from summary.json; it is representative of this model's quality. Row-max in bold. This repo = v7-coderx.

Benchmark 128e (unpruned) v6-coder v7-coder v7-coderx
GPQA-diamond (198q) 67.17 61.11 51.52 51.01
AIME (30q) 73.33 56.67 80.00 76.67
MATH500 (100q) 92.00 89.00 95.00 95.00
GSM8K (100q) 89.00 88.00 91.00 93.00
ARC-Challenge (full) 96.50 95.39 92.15 86.60
IFEval (100q, strict) 97.00 92.00 92.00 92.00
HumanEval (164) 97.56 98.17 98.17 96.95
HumanEval+ (164) 92.07 92.68 92.07 93.29
LCB-medium-55 v4 96.36 92.73 98.18 92.73
LCB-medium-100 v4 97.00 94.00 94.00 91.00
MultiPL-E (100) 90.00 89.00 89.67 89.00

Metrics: GPQA & GSM8K = exact_match flexible-extract ยท MATH500 = math_verify ยท ARC & AIME = exact_match ยท IFEval = prompt_level_strict_acc ยท HumanEval/+ = pass@1 chat-extract ยท LCB-55/100 & MultiPL-E = pass@1. 128e uses the lcb_medium_55/100 templates; the prunes use lcb_medium_*_v4 (corrected harness, equivalent task). The all-hard LCB-77 cross-model comparison is the discriminating code slice (v7-coderx 85.71%, cohort-best).

Recipe (summary)

98e prune from 128e via the code4/lcb3 recipe (generate_drop_map_v5fk: generic_code 4ร—, targeted_lcb_medium_55 3ร—, no per-layer floor clamp), then the agentic loop-protection force-keep (46 experts) and the mandatory shared-FFN ฮฑ=1.2 upweight, then NVFP4A16 quantization. This is the loop-fixed build โ€” it replaces the earlier looping fs2440 prune. v7-coderx is the code-maximal sibling of v7-coder. Full recipe and calibration-class table are on the bf16 card.

Intended use & limitations

A compact, vLLM-deployable Gemma 4 checkpoint for maximal coding throughput and instruction-following โ€” the code-extreme (x) member of the v7-coder cohort. For the broader LCB-medium lead and HumanEval, use v7-coder, which leads those slices (GPQA โ‰ˆ 51 on both siblings โ€” neither recovers science). A research prune, not an official Google release; generic_multilingual is de-weighted (0ร—) and graduate science (GPQA) is a budget axis โ€” at 51.01% it is well below the unpruned 128e (67.17%). For llama.cpp/CPU deployment use the GGUF repo.

Lineage

128e โ†’ (v4 โ†’ v5 โ†’ v6-coder code line) โ†’ v7 competence-map rebuild โ†’ code4/lcb3 selection + agentic loop-protection force-keep = v7-coderx โ†’ NVFP4A16. Built and evaluated on the omnimergekit toolchain.

Downloads last month
24
Safetensors
Model size
11B params
Tensor type
BF16
ยท
U8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ManniX-ITA/gemma-4-A4B-98e-v7-coderx-NVFP4A16

Quantized
(3)
this model

Collection including ManniX-ITA/gemma-4-A4B-98e-v7-coderx-NVFP4A16