Instructions to use SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M # Run inference directly in the terminal: llama cli -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M # Run inference directly in the terminal: llama cli -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M # Run inference directly in the terminal: ./build/bin/llama cli -hf SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
Use Docker
docker model run hf.co/SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF with Ollama:
ollama run hf.co/SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
- Unsloth Desktop
- Docker Model Runner
How to use SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF with Docker Model Runner:
docker model run hf.co/SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
- Lemonade
How to use SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull SyndicateLabs/Qwen3.8-27B-OBLITERATED-GGUF:IQ1_M
Run and chat with the model
lemonade run user.Qwen3.8-27B-OBLITERATED-GGUF-IQ1_M
List all available models
lemonade list
- Atomic Chat
Qwen3.8-27B OBLITERATED β The Pocket-Quant Ladder
A 27B dense hybrid (16 full-attention + 49 gated-delta SSM layers) with its MTP draft head intact, squeezed into the two smallest careful quants we could build. One 24 GB card runs the IQ1_M with room for context. Self-speculating via draft-mtp β no separate draft model needed.
πͺ The Ladder
| File | Size | ~bpw eff | MTP block | Min rig |
|---|---|---|---|---|
Qwen3.8-27B-OBLITERATED-IQ2M.gguf |
12.0 GB | ~3.5 | β | 1Γ RTX 4090 / 5090 (24 GB) |
Qwen3.8-27B-OBLITERATED-IQ1M.gguf |
10.2 GB | ~2.9 | β | 1Γ 3090/4090/5090 (24 GB, tons of ctx) |
π§ The Tensor Budget
Dense is the hardest quant class β no expert redundancy to hide behind. The budget:
| Tensor | Type | Why |
|---|---|---|
token_embd |
Q8_0 | every token |
output |
Q6_K | logits |
attention (attn_q/k/v/output) |
Q5_K / Q6_K | 16 full-attn layers, protected |
ssm* (gated-delta SSM layers) |
Q5_K | 49 layers of dynamics β the model's memory |
blk.64 (MTP block) |
Q8_0 | draft head exact β speculation acceptance stays high |
FFN (ffn_gate/down) |
body ftype | the bulk |
π Serving (llama.cpp β₯ v0.4.1)
llama-server \
-m Qwen3.8-27B-OBLITERATED-IQ2M.gguf \
-ngl 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
--spec-type draft-mtp --spec-draft-n-max 2 \
--jinja --ctx-size 32768
The MTP block (blk.64) was pinned at Q8_0 through the entire chain β draft quality preserved, acceptance rate stays honest. Requires a build with qwen3_5 hybrid support (llama.cpp β₯ mid-Sept 2026 master / v0.4.1, includes the GDN normalization fix).
π Provenance
- Source:
OBLITERATUS/Qwen3.8-27B-OBLITERATED(BF16, 55.6 GB, 1.2M-download abliteration) - Master: converted to Q8_0 with the MTP head kept and verified
- imatrix: fresh, 32 chunks, security-weighted corpus
- Ladder: IQ2_M + IQ1_M dynamic requants, uploaded then cleaned
π Credits
- OBLITERATUS β the source obliteration
- The Qwen3.5 hybrid-attention lineage (gated delta net)
- The llama.cpp imatrix / i-quant ecosystem
β οΈ Responsible Use
Obliterated/uncensored: intended for red-teaming, security research, evaluation, and unfiltered assistant work. You are responsible for what you do with it. Don't do crimes.
- Downloads last month
- 1,943
1-bit
2-bit