Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog

TL;DR

This work tackles the high resource demands of large language models by introducing Prune-Tune Loop (PTL), a gradual, iterative framework that prunes redundant reasoning parameters (neurons and layers) and performs lightweight recovery via continual pre-training or reinforcement learning. By dividing compression into fine-grained iterations, PTL avoids abrupt performance losses and demonstrates robust reasoning maintenance across models like Llama3-8B, Gemma2-9B, and Qwen2.5-7B, achieving roughly a 30–40% reduction in parameters and significant FLOP speedups with minimal loss on math benchmarks and even improved performance on some tasks. The method emphasizes structured pruning, stable pruning order, and efficient recovery, and extends to coding tasks such as MBPP, indicating broad applicability beyond mathematical reasoning. Overall, PTL presents a practical, modular approach to efficient LLM deployment with strong reasoning fidelity and broad transferability to diverse tasks.

Abstract

Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, but their substantial size often demands significant computational resources. To reduce resource consumption and accelerate inference, it is essential to eliminate redundant parameters without compromising performance. However, conventional pruning methods that directly remove such parameters often lead to a dramatic drop in model performance in reasoning tasks, and require extensive post-training to recover the lost capabilities. In this work, we propose a gradual compacting method that divides the compression process into multiple fine-grained iterations, applying a Prune-Tune Loop (PTL) at each stage to incrementally reduce model size while restoring performance with finetuning. This iterative approach-reminiscent of the "boiling frog" effect-enables the model to be progressively compressed without abrupt performance loss. Experimental results show that PTL can compress LLMs to nearly half their original size with only lightweight post-training, while maintaining performance comparable to the original model on reasoning tasks. Moreover, PTL is flexible and can be applied to various pruning strategies, such as neuron pruning and layer pruning, as well as different post-training methods, including continual pre-training and reinforcement learning. Additionally, experimental results confirm the effectiveness of PTL on a variety of tasks beyond mathematical reasoning, such as code generation, demonstrating its broad applicability.

Figures (5)

• Figure 1: Prune–Tune Loop (PTL) mainly consists of three steps in each minimal iteration: ① identify redundant reasoning parameters, including either redundant neurons or layers; ② prune the identified redundant reasoning parameters; ③ tune the pruned model through either continual pre-training on CoT data or reinforcement learning using complex mathematical datasets. By introducing only minor changes at each iteration, performance degradation remains minimal and quickly recoverable, avoiding abrupt disruptions like the "boiling frog" effect.
• Figure 2: PTL-Gemma2-5B generates more structured CoT and avoids producing meaningless output.
• Figure 3: PTL-Gemma2-9B’s performance over a complete pruning cycle
• Figure 4: The impact of varying pruning step size on the model’s final performance.
• Figure : Prune–Tune Loop