Demystifying When Pruning Works via Representation Hierarchies
Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network pruning from a representation-hierarchy perspective, decomposing the internal computation of language models into three sequential spaces: embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the embedding and logit spaces are largely robust to pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-token probability subspace, together with the robustness of the embedding space, supports the effectiveness of pruning for non-generative tasks such as retrieval and multiple-choice selection. Our analysis disentangles the effects of pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/Pruning-on-Representations
69d5b92ab9b236b710ae80c4
Shwai He
claimed_verified
2026-04-13T07:34:51.649Z
64468d7a3b28db22e7f84418
/avatars/ff05994acd7da64a2ed6e44f12cf77bc.svg
Shwai He
shwai-he
user
shwai-he
69d5b92ab9b236b710ae80c5
Guoheng Sun
69d5b92ab9b236b710ae80c6
Haichao Zhang
claimed_verified
2026-04-17T06:59:22.414Z
66393f5a1231260674ae798e
https://cdn-avatars.huggingface.co/v1/production/uploads/66393f5a1231260674ae798e/JlWLdLTjMwgh0x1ASx4Mx.jpeg
Haichao Zhang
haichaozhang
user
haichaozhang
69d5b92ab9b236b710ae80c7
Yun Fu
69d5b92ab9b236b710ae80c8
Ang Li
Demystifying When Pruning Works via Representation Hierarchies
Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in generative settings. To understand this discrepancy, we analyze network pruning from a representation-hierarchy perspective, decomposing the internal computation of language models into three sequential spaces: embedding (hidden representations), logit (pre-softmax outputs), and probability (post-softmax distributions). We find that representations in the embedding and logit spaces are largely robust to pruning-induced perturbations. However, the nonlinear transformation from logits to probabilities amplifies these deviations, which accumulate across time steps and lead to substantial degradation during generation. In contrast, the stability of the categorical-token probability subspace, together with the robustness of the embedding space, supports the effectiveness of pruning for non-generative tasks such as retrieval and multiple-choice selection. Our analysis disentangles the effects of pruning across tasks and provides practical guidance for its application. Code is available at https://github.com/CASE-Lab-UMD/Pruning-on-Representations
Network pruning affects different representation spaces differently, leading to varying performance across tasks due to instability in probability space transformations during generation.
network pruning representation-hierarchy embedding space logit space probability space categorical-token probability subspace generative settings non-generative tasks pruning-induced perturbations