Inkling-2-Mini-GGUF

GGUF quantizations of thinkingmachines/Inkling-2-Mini, converted and quantized with Hearth by Interchained.

โš ๏ธ Runtime requirement

Inkling 2 uses a new looped architecture (inkling2_loop). These files require a llama.cpp build with Inkling 2 support โ€” ggml-org/llama.cpp PR #32695, which is merged into the aiassistsecure/llama.cpp fork master. Stock llama.cpp releases without that PR will not load these files.

The model

Inkling 2 Mini is Thinking Machines Lab's general-purpose multimodal model: a 24-layer decoder-only transformer (26.4B parameters, ~3.4B active per token) whose middle 12 layers run three times with shared weights โ€” 48 layer applications per token from 24 layers of weights. Each token is routed to 8 of 128 MoE experts; attention mixes local (1024-token window) and global layers; native context is 1M tokens. Released by Thinking Machines under Apache-2.0 in mixed NVFP4/MXFP8 precision; these GGUFs were produced from that official checkpoint.

Scope of this release: text inference. The vision projector (mmproj) is not included in this release, so image input is not enabled in these files.

Quants

File Size BPW Notes
Inkling-2-Mini-q8_0.gguf 25.83 GiB 8.51 Highest fidelity quant
Inkling-2-Mini-q6_k.gguf 21.80 GiB 7.18
Inkling-2-Mini-q5_k_m.gguf 18.45 GiB 6.08
Inkling-2-Mini-q4_k_m.gguf 16.19 GiB 5.33 Recommended โ€” fits a 24 GB card with room for context
Inkling-2-Mini-q4_0.gguf 13.81 GiB 4.55
Inkling-2-Mini-q3_k_m.gguf 12.69 GiB 4.18
Inkling-2-Mini-q2_k.gguf 10.02 GiB 3.30
Inkling-2-Mini-tq1_0.gguf 8.25 GiB 2.72 Ternary (1-bit class) โ€” smallest, expect quality loss

Total parameters: 26.4B. On a 32 GB card (e.g. Tesla V100 32GB), every quant above fits fully in VRAM.

Quantization note

The expert down-projection tensors have 1152 rows, which K-quant block sizes cannot divide. In the K-quants, llama.cpp's quantizer therefore assigns fallback types to 120 of 694 tensors (Q5_0/Q4_0-class types on those tensors). This is the quantizer's standard behavior for this architecture and is reflected in the effective BPW figures above.

Usage

With a llama.cpp build that includes Inkling 2 support:

llama-server -m Inkling-2-Mini-q4_k_m.gguf -c 8192 --port 8080

or via Hearth:

hearth serve --model Inkling-2-Mini --gguf Inkling-2-Mini-q4_k_m.gguf

Credits

Downloads last month
-
GGUF
Model size
26B params
Architecture
inkling2
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Interchained/Inkling-2-Mini-GGUF

Quantized
(143)
this model