On the Limits of Layer Pruning for Generative Reasoning in LLMs
Recent works have shown that layer pruning can compress large language models (LLMs) while retaining strong performance on classification benchmarks with little or no finetuning. However, existing pruning techniques often suffer severe degradation on generative reasoning tasks. Through a systematic study across multiple model families, we find that tasks requiring multi-step reasoning are particularly sensitive to depth reduction. Beyond surface-level text degeneration, we observe degradation of critical algorithmic capabilities, including arithmetic computation for mathematical reasoning and balanced parenthesis generation for code synthesis. Under realistic post-training constraints, without access to pretraining-scale data or compute, we evaluate a simple mitigation strategy based on supervised finetuning with Self-Generated Responses. This approach achieves strong recovery on classification tasks, retaining up to 90\% of baseline performance, and yields substantial gains of up to 20--30 percentage points on generative benchmarks compared to prior post-pruning techniques. Crucially, despite these gains, recovery for generative reasoning remains fundamentally limited relative to classification tasks and is viable primarily at lower pruning ratios. Overall, we characterize the practical limits of layer pruning for generative reasoning and provide guidance on when depth reduction can be applied effectively under constrained post-training regimes.
6981a43ece18b18628096410
Safal Shrestha
claimed_verified
2026-02-03T08:22:43.269Z
64cb922ec7f30fbf7b91a9a7
/avatars/457eae5e56b9641ee5543146447d1755.svg
Safal Shrestha
safal312
user
safal312
6981a43ece18b18628096411
Anubhav Shrestha
6981a43ece18b18628096412
Aadim Nepal
6981a43ece18b18628096413
Minwu Kim
claimed_verified
2026-02-03T10:02:29.596Z
64e77b47d96966317b45eeb3
/avatars/6b67eba3f15d6cd86ac3ad55c1daf166.svg
Minwu Kim
guactastesgood
user
guactastesgood
6981a43ece18b18628096414
Keith Ross
On the Limits of Layer Pruning for Generative Reasoning in LLMs
Recent works have shown that layer pruning can compress large language models (LLMs) while retaining strong performance on classification benchmarks with little or no finetuning. However, existing pruning techniques often suffer severe degradation on generative reasoning tasks. Through a systematic study across multiple model families, we find that tasks requiring multi-step reasoning are particularly sensitive to depth reduction. Beyond surface-level text degeneration, we observe degradation of critical algorithmic capabilities, including arithmetic computation for mathematical reasoning and balanced parenthesis generation for code synthesis. Under realistic post-training constraints, without access to pretraining-scale data or compute, we evaluate a simple mitigation strategy based on supervised finetuning with Self-Generated Responses. This approach achieves strong recovery on classification tasks, retaining up to 90\% of baseline performance, and yields substantial gains of up to 20--30 percentage points on generative benchmarks compared to prior post-pruning techniques. Crucially, despite these gains, recovery for generative reasoning remains fundamentally limited relative to classification tasks and is viable primarily at lower pruning ratios. Overall, we characterize the practical limits of layer pruning for generative reasoning and provide guidance on when depth reduction can be applied effectively under constrained post-training regimes.
Layer pruning compresses large language models while maintaining classification performance but causes significant degradation in generative reasoning tasks, with limited recovery possible through supervised finetuning on self-generated responses.
layer pruning large language models classification benchmarks generative reasoning tasks multi-step reasoning arithmetic computation balanced parenthesis generation post-training supervised finetuning Self-Generated Responses