When Does Sparsity Mitigate the Curse of Depth in LLMs
Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.
69b91c5b6b3f36c249fbc677
Dilxat Muhtar
claimed_verified
2026-03-17T09:36:31.670Z
631551858be84e1ab41478f4
/avatars/c464f009ec0b7ec8e144ab8fbd07de43.svg
Dilxat Muhtar
PumpkinCat
user
PumpkinCat
69b91c5b6b3f36c249fbc678
Xinyuan Song
admin_assigned
2026-03-17T09:56:47.078Z
6728a79b2a2e92f37afb900d
/avatars/92106187ccbe34dfdbfc5d6d6fd63210.svg
XinyuanSong
XinyuanSong
user
XinyuanSong
69b91c5b6b3f36c249fbc679
Sebastian Pokutta
69b91c5b6b3f36c249fbc67a
Max Zimmer
69b91c5b6b3f36c249fbc67b
Nico Pelleriti
admin_assigned
2026-03-17T09:57:34.723Z
67dfd361d33e17f3fee90455
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zaOc8X1ij9C_X-D4P5Us_.png
Nico Pelleriti
n-pelleriti
user
n-pelleriti
69b91c5b6b3f36c249fbc67c
Thomas Hofmann
69b91c5b6b3f36c249fbc67d
Shiwei Liu
When Does Sparsity Mitigate the Curse of Depth in LLMs
Recent work has demonstrated the curse of depth in large language models (LLMs), where later layers contribute less to learning and representation than earlier layers. Such under-utilization is linked to the accumulated growth of variance in Pre-Layer Normalization, which can push deep blocks toward near-identity behavior. In this paper, we demonstrate that, sparsity, beyond enabling efficiency, acts as a regulator of variance propagation and thereby improves depth utilization. Our investigation covers two sources of sparsity: (i) implicit sparsity, which emerges from training and data conditions, including weight sparsity induced by weight decay and attention sparsity induced by long context inputs; and (ii) explicit sparsity, which is enforced by architectural design, including key/value-sharing sparsity in Grouped-Query Attention and expert-activation sparsity in Mixtureof-Experts. Our claim is thoroughly supported by controlled depth-scaling experiments and targeted layer effectiveness interventions. Across settings, we observe a consistent relationship: sparsity improves layer utilization by reducing output variance and promoting functional differentiation. We eventually distill our findings into a practical rule-of-thumb recipe for training deptheffective LLMs, yielding a notable 4.6% accuracy improvement on downstream tasks. Our results reveal sparsity, arising naturally from standard design choices, as a key yet previously overlooked mechanism for effective depth scaling in LLMs. Code is available at https://github.com/pUmpKin-Co/SparsityAndCoD.