From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models

TL;DR

This work tackles the cost of large language models by examining pruning strategies across instruction-following (LLM-instruct) and reasoning-augmented (LLM-think) paradigms. It introduces a controlled pruning pipeline that aligns calibration and post-pruning recovery data with each model’s training distribution, and evaluates static depth, static width, and dynamic pruning on 17 tasks spanning classification, generation, and reasoning. The results reveal paradigm- and task-dependent trade-offs: static depth pruning excels at classification, static width pruning better supports generation and reasoning, and dynamic pruning helps in some tasks but struggles with long-chain reasoning, highlighting the need for reasoning-aware pruning methods. The findings emphasize that effective pruning for reasoning-augmented LLMs requires strategies that preserve continuous computation along depth and account for longer intermediate traces, with practical implications for deploying efficient reasoning-enabled systems; code and datasets are publicly available.

Abstract

Large language models (LLMs) are increasingly costly to deploy, motivating extensive research on model pruning. However, most existing studies focus on instruction-following LLMs, leaving it unclear whether established pruning strategies transfer to reasoning-augmented models that explicitly generate long intermediate reasoning traces. In this work, we conduct a controlled study of pruning for both instruction-following ($\textbf{LLM-instruct}$) and reasoning-augmented ($\textbf{LLM-think}$) models. To isolate the effects of pruning, we align pruning calibration and post-pruning recovery data with each model's original training distribution, which we show yields more stable and reliable pruning behavior. We evaluate static depth pruning, static width pruning, and dynamic pruning across 17 tasks spanning classification, generation, and reasoning. Our results reveal clear paradigm-dependent differences: depth pruning outperforms width pruning on classification tasks, while width pruning is more robust for generation and reasoning. Moreover, static pruning better preserves reasoning performance, whereas dynamic pruning excels on classification and generation but remains challenging for long-chain reasoning. These findings underscore the need for pruning strategies that explicitly account for the distinct characteristics of reasoning-augmented LLMs. Our code is publicly available at https://github.com/EIT-NLP/LRM-Pruning.

Figures (10)

• Figure 1: Overview of the three structured pruning strategies. Static depth pruning removes entire layers, static width pruning reduces hidden dimensions (neurons or attention heads), and dynamic depth pruning adaptively skips layers, attention blocks, or MLP modules depending on the input.
• Figure 2: Experimental results validating the impact of calibration and post-training datasets (left: LLM-instruct; right: LLM-think), with results averaged across SliceGPT and ShortGPT.
• Figure 3: Training loss of LLM-instruct at 20% pruning ratio. Dynamic methods exhibit substantially lower loss (with D-LLM as an exception).
• Figure 4: Performance of different pruning methods under varying pruning ratios on classification, generation, and reasoning tasks.
• Figure 5: Training loss of LLM-think at 20% pruning ratio. Dynamic methods exhibit substantially higher loss (with MOD as an exception), whereas static depth and width pruning remain lower and more stable.