mradermacher

/

model_requests

like 187

Hi! You can significantly crop model size by removing all non-latin tokens. Check this pip package prunetokens

Qwen3.5-9B-AWQ result: 8.5 GB → 7.0 GB

We have the model quanting automated so i don't know how we would even implement it, and some people want to have non latin characters

1. we dont do awq
2. why would we remove all non-latin tokens? what about our amazing friends that dont use latin alphabet? Like you know, 70%+ of the world?

1. we dont do awq

AWQ is just an example. The tool works with .safetensors.

2. why would we remove all non-latin tokens? what about our amazing friends that dont use latin alphabet? Like you know, 70%+ of the world?

But you can get a smaller model—with fewer requirements and faster inference.

1. we dont do awq

AWQ is just an example. The tool works with .safetensors.

2. why would we remove all non-latin tokens? what about our amazing friends that dont use latin alphabet? Like you know, 70%+ of the world?

But you can get a smaller model—with fewer requirements and faster inference.

Yes, then if you only care about english, do the quants yourself

But you can get a smaller model—with fewer requirements and faster inference.

yes, but we dont want to confuse the future users. We already have somewhat confused people while having rules such as "we quant using only official main branch of llamacpp", so using prunetokens to remove the tokens that 70% of the world cant use... it's bad