Instructions to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid

SGLang

How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid with Docker Model Runner:
```
docker model run hf.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid
```

GLM-5.2-Vision NVFP4+AQLM hybrid — image input + ~1M context on 4x RTX PRO 6000 (SM120, 384 GB)

Baseten's GLM-5.2-Vision graft on the NVFP4+AQLM hybrid quant — a 744B-param GLM-5.2 that now takes images, still serving a ~950K-token window on 4x 96 GB SM120 GPUs with lossless MTP speculative decode and disk-backed KV persistence.

Credits

Baseten built the vision graft (baseten/GLM-5.2-Vision-NVFP4): the MoonViT vision encoder from Moonshot AI's Kimi-K2.6 bolted onto Z.ai's GLM-5.2 through a PatchMerger projector baseten trained (49.5M params, 1152→4608→6144) — without touching a single GLM weight. The vision tower (MoonViT-3d, 27 layers, 1152-dim, ~417M params) and projector in this repo are byte-identical BF16 copies of baseten's release.
Z.ai — GLM-5.2 (MIT). Moonshot AI — Kimi-K2.6 / MoonViT (Modified MIT).
This repo swaps baseten's frozen NVFP4 text backbone for our NVFP4+AQLM hybrid quant (292 GB, all 256 experts routable), which is what makes the ~1M window fit on 4x 96 GB instead of 8x B200.

Quickstart (4x RTX PRO 6000, SM120)

Not stock vLLM — use the pre-patched fork, and note the branch: vision support lives on vision-graft, not the text-only cudagraphs-v2 branch.

git clone -b vision-graft https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120 && git checkout 57955eed9   # exact gated head
docker build -f Dockerfile.glm52-sm120 -t glm52-vision-sm120 .

docker run --gpus all --ipc=host -p 8001:8001 \
  -v /path/to/this-repo:/models/1m:ro \
  -e PARALLEL=tp4-1m-mtp \
  -e ENABLE_LMCACHE=1 -e UTIL=0.96 \
  glm52-vision-sm120 --trust-remote-code

Three things differ from the text-only card and all three are required:

Branch vision-graft (head 57955eed9, built on cudagraphs-v2). It adds the glm5v architecture (Glm5vForConditionalGeneration): MoonViT tower + projector wired into the fork's in-tree Kimi-K2.5 vision code, with the language model swapped to the hybrid GlmMoeDsaForCausalLM stack (DCP + native MTP + SM120 sparse-MLA fixes + NVFP4/AQLM MoE kernels). The text-only branch cannot load this repo's config.
--trust-remote-code — image preprocessing uses the Kimi-K2.5-style processor files shipped in this repo (kimi_k25_processor.py etc.).
UTIL=0.96 when LMCache is on — measured KV pool 991,727 tokens ≥ the 950K window. The entrypoint's LMCache default of 0.95 lands ~917K (boot failure); 0.97 leaves too little headroom for LMCache's ~1 GiB/GPU staging buffer and risks store-OOM. With ENABLE_LMCACHE=0 the default 0.97 is fine (KV 1,066,222).

The image build pins flashinfer to a known-good commit and stubs nvidia-cutlass-dsl's cute.experimental (the republished 4.5.2 wheel AttributeErrors at import flashinfer.comm), so a plain docker build is index-independent and reproducible.

Then send images through the standard OpenAI API:

curl http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "glm-5.2",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "Describe this image."}
  ]}]
}'

Validation (2026-07-23, 4x RTX PRO 6000)

Everything was gated combined — vision + 950K window + MTP + LMCache in one serving config — against the production text-only build as reference:

Text regression: prompts without images produce output identical to the text-only production build (the graft leaves the text path untouched).
Vision: single- and two-image prompts described correctly with correct per-image attribution, no hallucination; re-verified live on the exact baked production image.
Long context + vision: needle-in-haystack with an image in context, exact at 130K tokens.
MTP: acceptance 3.17–3.30 accepted/step (72–77%) with images present — inside the healthy envelope of the text-only production stack; spec decode stays lossless by rejection sampling.
LMCache: cold-restart disk restore of a long mixed prompt in 3.4s vs 109s recompute (~32x), byte-identical output vs the cache-miss path.
Cost of sight: tower+projector add ~~320 MB/GPU at TP4 (~~58K tokens of KV pool) — the 950K window still fits, at util 0.96 even with LMCache's buffers.

Bare launch — equivalent flags without the docker entrypoint

export NCCL_MAX_NCHANNELS=4 NCCL_BUFFSIZE=1048576 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192
export NCCL_P2P_LEVEL=SYS VLLM_MTP_INDEX_SHARE=1 GLM_MOE_LANE_ROWS=1 GLM_NVFP4_LUT256=1
vllm serve <repo-dir> --trust-remote-code \
  --tensor-parallel-size 4 \
  --decode-context-parallel-size 4 --dcp-comm-backend ag_rs \
  --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}' \
  --compilation-config '{"mode":3,"cudagraph_mode":"PIECEWISE"}' \
  --gpu-memory-utilization 0.97 --kv-cache-dtype fp8_ds_mla \
  --max-model-len 950000 --max-num-seqs 2 --max-num-batched-tokens 4096 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --served-model-name glm-5.2 --port 8001

This bypasses the entrypoint, so no LMCache (use the docker run form for disk-backed KV; that path needs the patched jarrelscy/LMCache@glm52-dcp-dsa fork and util 0.96). cudagraph_mode must stay PIECEWISE for DCP+spec; do not add --enforce-eager.

Quantization & performance

The text backbone is exactly jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid (hot ~30% of experts NVFP4 @4.5 bpw, cold experts PV-tuned AQLM @2 bpw, attention/DSA/dense/shared/embeddings BF16, fp8_ds_mla KV) — see that card for the scheme, teacher-forced perplexity vs. other quants, GPQA Diamond 0.889, and measured throughput; text-path numbers carry over unchanged. Vision tower and projector run in BF16 (tower sharded across TP ranks, projector replicated).

Terminal-Bench 2.1

Agentic-coding eval on this exact serving stack (the hybrid quant above; text path, no images), using the harness's terminus-2 agent driving glm-5.2 over the OpenAI API at reasoning_effort: max, temperature: 1:

metric	score
pass@1	78.4% (69 / 88 resolved)

pass@1 = the earliest-completed attempt per task, scored by the task's own verifier. 88 of the 89 tasks reached a verdict; 1 task (regex-chess) is excluded from the denominator because the agent never converged to a submission — it looped on tool-call exploration for hours without writing a solution file, so no verdict exists to count either way. Counting it as a failure instead gives 69/89 = 77.5%.
The run was executed in two halves against the same model weights (an initial 47-task pass, then the remaining 42 re-run on the local 4x RTX PRO 6000 endpoint) and combined under one consistent pass@1 rule; each half scored ~78% independently (37/47 and 32/41).
Per-task trajectories, verifier outputs, and the results table are published under traces/ in this repo.

Video input is untested. For text-only serving, the text card and its cudagraphs-v2 branch remain available, though this repo is a strict superset in capability.

Downloads last month: 206

Safetensors

Model size

208B params

Tensor type

BF16

F32

F16

I16

Model tree for jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid

Base model

zai-org/GLM-5.2

Quantized

baseten/GLM-5.2-Vision-NVFP4

Quantized

(8)

this model

Quantizations

1 model