When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

TL;DR

This paper investigates how layer pruning affects long-chain reasoning in LLMs by using test-time scaling as a lens to probe reasoning robustness beyond standard accuracy. It systematically evaluates sequential and parallel scaling under pruning across two models and multiple pruning strategies, showing that sequential scaling is highly fragile while merging-based pruning can help preserve parallel scaling. Fine-tuning methods (LoRA and full-parameter) do not fully recover the degraded sequential scaling, pointing to fundamental structural impacts on reasoning dynamics. The authors provide in-depth analyses of reasoning loops, diversity loss, and self-reflection deterioration, arguing for pruning approaches that safeguard inference-time reasoning capabilities and proposing avenues for robust, efficiency-conscious pruning design.

Abstract

Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs). Although existing methods demonstrate strong performance retention on general knowledge tasks, their effect on long-chain reasoning, a more brittle yet crucial capability, remains largely unexplored. In this work, we study the impact of layer pruning on long-chain reasoning through the lens of test-time scaling, a key mechanism in modern LLMs that enables strong reasoning capacity by allocating more computation at inference time. With extensive experiments, we demonstrate that pruning even one or two layers can severely impair test-time scaling, with performance collapsing drastically on long reasoning benchmarks even when performance on knowledge-intensive and shallow reasoning tasks remains stable. Furthermore, we find that standard supervised fine-tuning remedies fail to recover test-time scaling once it has deteriorated. Through in-depth analyses, we identify the mechanisms underlying this fragility of test-time scaling and highlight the fundamental risks of applying layer pruning to reasoning-intensive LLMs. These findings call for a rethinking of layer pruning strategies and provide insights for developing methods that preserve the robustness of reasoning. We open-source the codebase in \href{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}.

Figures (14)

• Figure 1: Accuracy of the pruned Qwen3-8B by ShortGPT across various pruning depths on MMLU, GSM8K, MATH500 and AIME24. LLM-as-a-Judge is applied to MATH500 and AIME24 as described in Section \ref{['section3']}.
• Figure 2: Sequential test-time scaling of s1.1-7B under different pruning depths.
• Figure 3: Parallel test-time scaling of s1.1-7B on AIME24 under different pruning depths.
• Figure 4: Sequential test-time scaling of s1.1-7B after Full FT under different pruning depths. We denote "Full" as Full FT and "Reverse" as Reverse-order in the figure for simplicity.
• Figure 5: Examples of repetitive thinking from 1-layer pruned s1.1-7B model. Unlike the original model, the pruned version gets stuck in logic loops. The teal text denotes the correct reasoning trace while light blue text is the repetitive reasoning trace leading to failure, and the blue text is a brief summary of the failure reason.