Qwen3.8-27B OBLITERATED pocket-quant ladder

Qwen3.8-27B OBLITERATED β€” The Pocket-Quant Ladder

A 27B dense hybrid (16 full-attention + 49 gated-delta SSM layers) with its MTP draft head intact, squeezed into the two smallest careful quants we could build. One 24 GB card runs the IQ1_M with room for context. Self-speculating via draft-mtp β€” no separate draft model needed.

πŸͺœ The Ladder

File Size ~bpw eff MTP block Min rig
Qwen3.8-27B-OBLITERATED-IQ2M.gguf 12.0 GB ~3.5 β€” 1Γ— RTX 4090 / 5090 (24 GB)
Qwen3.8-27B-OBLITERATED-IQ1M.gguf 10.2 GB ~2.9 β€” 1Γ— 3090/4090/5090 (24 GB, tons of ctx)

🧠 The Tensor Budget

Dense is the hardest quant class β€” no expert redundancy to hide behind. The budget:

Tensor Type Why
token_embd Q8_0 every token
output Q6_K logits
attention (attn_q/k/v/output) Q5_K / Q6_K 16 full-attn layers, protected
ssm* (gated-delta SSM layers) Q5_K 49 layers of dynamics β€” the model's memory
blk.64 (MTP block) Q8_0 draft head exact β†’ speculation acceptance stays high
FFN (ffn_gate/down) body ftype the bulk

πŸš€ Serving (llama.cpp β‰₯ v0.4.1)

llama-server \
  -m Qwen3.8-27B-OBLITERATED-IQ2M.gguf \
  -ngl 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --jinja --ctx-size 32768

The MTP block (blk.64) was pinned at Q8_0 through the entire chain β€” draft quality preserved, acceptance rate stays honest. Requires a build with qwen3_5 hybrid support (llama.cpp β‰₯ mid-Sept 2026 master / v0.4.1, includes the GDN normalization fix).

πŸ“œ Provenance

  1. Source: OBLITERATUS/Qwen3.8-27B-OBLITERATED (BF16, 55.6 GB, 1.2M-download abliteration)
  2. Master: converted to Q8_0 with the MTP head kept and verified
  3. imatrix: fresh, 32 chunks, security-weighted corpus
  4. Ladder: IQ2_M + IQ1_M dynamic requants, uploaded then cleaned

πŸ™ Credits

  • OBLITERATUS β€” the source obliteration
  • The Qwen3.5 hybrid-attention lineage (gated delta net)
  • The llama.cpp imatrix / i-quant ecosystem

⚠️ Responsible Use

Obliterated/uncensored: intended for red-teaming, security research, evaluation, and unfiltered assistant work. You are responsible for what you do with it. Don't do crimes.

Downloads last month
1,943
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF

Base model

Qwen/Qwen3.8-27B
Quantized
(24)
this model