Any findings? 😀

@sousekd

Thanks for the benchmarks! I'm back at my desk today and gonna do some more perplexity measures, but you can see in the updated graph that the full q8_0 everything is scoring "better" than the q8_0-q4_0 which I wouldn't expect if the QAT was completely effective maybe?

Some folks on ai beavers discord want to measure the original with VLLM and try to get apples-apples comparison with a GGUF but that is tricky to do especially with these large sizes.

The main thing I've seen is to run this with an updated chat template (given the original was patched a couple times since release) as well as adding --special to get it to output <think> tags but then you have to deal with the stop token in the client side..

I'm back at my desk today and gonna do some more perplexity measures, but you can see in the updated graph that the full q8_0 everything is scoring "better" than the q8_0-q4_0 which I wouldn't expect if the QAT was completely effective maybe?

Interesting. I know nothing about the quantization process, but I can imagine that converting any floating point number can lead to some data loss, so direct conversion would be preferred...

The main thing I've seen is to run this with an updated chat template [...]

...and the tool calling does not work yet - even on mainline llama.cpp.

...and the tool calling does not work yet - even on mainline llama.cpp.

Ohh really?? It seems like tool calling and MCP stuff can be so sensitive to exact implementation details, chat templates, and why Kimi released an entire tool K2-Vendor-Verifier github project to score a setup on how well it seems to actually be working...

Ohh really??

Yes: https://github.com/ggml-org/llama.cpp/issues/17155

I wanted to post a feature request on ik_llama, but it probably makes sense to wait for the llama.cpp implementation first.
There are more models with broken tool calling on ik_llama still. I hope to find a bit of time soon to test them all vs mainline and report back...

It is interesting how tool calling does not seem to be that important for many.
Even vLLM still has serious issues wuth gpt-oss tool calling - and it is not a new model by any means.
I guess most people use local models just for RP 😀. And coding, where the clients have their own implementation.

Just to compare with GLM 4.7 IQ5_K, here is a run of Kimi-K2 Thinking Q4_X with max supported context on RTX PRO 6000. It only fills 50% of VRAM (47 GB), while GLM 4.7 IQ5_K takes it all. This is without --grouped-expert-routing --merge-qkv, which I'm not quite sure what they are about:

Single RTX 6000, 256K context (max supported)

./llama-sweep-bench \
    --model "$MODEL_PATH" \
    --no-mmap \
    -mla 3 -amb 512 \
    -b 8192 -ub 8192 \
    -ctk f16 -ctv f16 -c 262144 \
    -ngl 999 -ncmoe 999 \
    --threads 24 \
    --threads-batch 32 \
    --warmup-batch \
    -n 256

GPU power capped at 450 W. CPU set to a “balanced”-style power profile in BIOS, also limited a bit.
Running in a Proxmox VM with only 32 physical CPU cores assigned, leaving the SMT threads for the host.
Full specs here for anyone interested.