On the Limits of Layer Pruning for Generative Reasoning in LLMs

TL;DR

This work analyzes the practical limits of layer pruning for generative reasoning in LLMs, revealing that generative tasks are far more sensitive to depth reduction than classification. It conducts a systematic, multi-model, layer-by-layer ablation and identifies failure modes in text generation, arithmetic, and syntax tracking, then evaluates Self-Generated Responses (SGR) as a simple post-training recovery under constrained data and compute. The results show that SGR improves classification retention and yields notable gains on generative benchmarks at moderate pruning, yet recovery for generative reasoning remains fundamentally limited compared to classification. The study provides concrete guidelines for when depth reduction is viable, advocating conservative pruning and task-aligned SGR strategies to balance efficiency with preserved generative capabilities.

Abstract

Recent works have shown that layer pruning can compress large language models (LLMs) while retaining strong performance on classification benchmarks with little or no finetuning. However, existing pruning techniques often suffer severe degradation on generative reasoning tasks. Through a systematic study across multiple model families, we find that tasks requiring multi-step reasoning are particularly sensitive to depth reduction. Beyond surface-level text degeneration, we observe degradation of critical algorithmic capabilities, including arithmetic computation for mathematical reasoning and balanced parenthesis generation for code synthesis. Under realistic post-training constraints, without access to pretraining-scale data or compute, we evaluate a simple mitigation strategy based on supervised finetuning with Self-Generated Responses. This approach achieves strong recovery on classification tasks, retaining up to 90\% of baseline performance, and yields substantial gains of up to 20--30 percentage points on generative benchmarks compared to prior post-pruning techniques. Crucially, despite these gains, recovery for generative reasoning remains fundamentally limited relative to classification tasks and is viable primarily at lower pruning ratios. Overall, we characterize the practical limits of layer pruning for generative reasoning and provide guidance on when depth reduction can be applied effectively under constrained post-training regimes.

Figures (14)

• Figure 1: Effect of removing a single layer on model performance across generative benchmarks. Reasoning-intensive tasks such as GSM8K and HumanEval+ exhibit severe performance degradation at specific layers, while XSUM remains comparatively robust except for layers whose removal induces general text degeneration. (We generally skip layer 0 for its poor results.)
• Figure 2: Text degeneration under single-layer pruning, measured using 4-gram repetition (left) and Self-BLEU4 averaged across responses and normalized relative to the baseline.
• Figure 3: Effect of single-layer pruning on the arithmetic ability of Llama.
• Figure 4: Distribution of syntactic error types induced by single-layer pruning.
• Figure 5: Perplexity curves during training for both standard finetuning and for SGR for the Llama model (BI pruned: 25%) Results for other models in \ref{['ppl-curves-appendix']}.