Model pruning for the masses!
text
Model pruning for the masses!
new_line
new_line
As of version [5740](
text
As of version [5740](
https://github.com/ggml-org/llama.cpp/releases/tag/b5740
https://github.com/ggml-org/llama.cpp/releases/tag/b5740
link
), 
text
), 
llama-quantize
`llama-quantize`
inline_code
 now supports layer pruning via the 
text
 now supports layer pruning via the 
--prune-layers
`--prune-layers`
inline_code
 flag!
text
 flag!
new_line
new_line
Findings so far are that removing one or two layers has a relatively moderate impact on quality. PPL and KLD suffer quite a lot, as expected considering that pruning changes the logits distribution, but the drop in inference quality, as reflected by tests' scores, is less pronounced.
text
Findings so far are that removing one or two layers has a relatively moderate impact on quality. PPL and KLD suffer quite a lot, as expected considering that pruning changes the logits distribution, but the drop in inference quality, as reflected by tests' scores, is less pronounced.
new_line
new_line
For example, using the Q4_K_M variants as a benchmark, the average drop between 
text
For example, using the Q4_K_M variants as a benchmark, the average drop between 
https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
eaddario/gemma-3-12b-it-pruned-GGUF
model
resource
https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF
 and 
text
 and 
https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
eaddario/gemma-3-12b-it-GGUF
model
resource
https://huggingface.co/eaddario/gemma-3-12b-it-GGUF
 is < 3% (60.03 vs 61.65). Similar behaviour for 
text
 is < 3% (60.03 vs 61.65). Similar behaviour for 
https://huggingface.co/eaddario/Qwen3-30B-A3B-pruned-GGUF
eaddario/Qwen3-30B-A3B-pruned-GGUF
model
resource
https://huggingface.co/eaddario/Qwen3-30B-A3B-pruned-GGUF
 and 
text
 and 
https://huggingface.co/eaddario/Qwen3-30B-A3B-GGUF
eaddario/Qwen3-30B-A3B-GGUF
model
resource
https://huggingface.co/eaddario/Qwen3-30B-A3B-GGUF
, albeit with a bit higher impact at ~5.5% (54.19 vs 57.36).
text
, albeit with a bit higher impact at ~5.5% (54.19 vs 57.36).
new_line
new_line
These results seem to confirm Xin Men's et al 
text
These results seem to confirm Xin Men's et al 
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)
https://huggingface.co/papers/2403.03853
2403.03853
paper
resource
https://huggingface.co/papers/2403.03853
new_line
new_line
Another interesting side-effect, at least with Qwen3-30B-A3B, is that pruning 3 or more layers makes the model forget English and reply in Chinese! but with still reasonable answers.
text
Another interesting side-effect, at least with Qwen3-30B-A3B, is that pruning 3 or more layers makes the model forget English and reply in Chinese! but with still reasonable answers.
Model pruning for the masses!

As of version [5740](https://github.com/ggml-org/llama.cpp/releases/tag/b5740), `llama-quantize` now supports layer pruning via the `--prune-layers` flag!

Findings so far are that removing one or two layers has a relatively moderate impact on quality. PPL and KLD suffer quite a lot, as expected considering that pruning changes the logits distribution, but the drop in inference quality, as reflected by tests' scores, is less pronounced.

For example, using the Q4_K_M variants as a benchmark, the average drop between https://huggingface.co/eaddario/gemma-3-12b-it-pruned-GGUF and https://huggingface.co/eaddario/gemma-3-12b-it-GGUF is < 3% (60.03 vs 61.65). Similar behaviour for https://huggingface.co/eaddario/Qwen3-30B-A3B-pruned-GGUF and https://huggingface.co/eaddario/Qwen3-30B-A3B-GGUF, albeit with a bit higher impact at ~5.5% (54.19 vs 57.36).

These results seem to confirm Xin Men's et al https://huggingface.co/papers/2403.03853

Another interesting side-effect, at least with Qwen3-30B-A3B, is that pruning 3 or more layers makes the model forget English and reply in Chinese! but with still reasonable answers.
🚀
John6666 AIMaster7 adjaysagar eugene-kamenev
669509fe388d30943970c4bf
/avatars/d7493113769fe72ac5391fb9d5de4b97.svg
Adjay Sagar
adjaysagar
user