Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoning

TL;DR

This work proposes Compress responses for Easy questions and Explore Hard ones (CEEH), a difficulty-aware approach to RL-based efficient reasoning that consistently reduces response length while maintaining accuracy comparable to the base model, and improves Pass@k relative to length-only optimization.

Abstract

Chain-of-Thought (CoT) has substantially empowered Large Language Models (LLMs) to tackle complex reasoning tasks, yet the verbose nature of explicit reasoning steps incurs prohibitive inference latency and computational costs, limiting real-world deployment. While existing compression methods - ranging from self-training to Reinforcement Learning (RL) with length constraints - attempt to mitigate this, they often sacrifice reasoning capability for brevity. We identify a critical failure mode in these approaches: explicitly optimizing for shorter trajectories triggers rapid entropy collapse, which prematurely shrinks the exploration space and stifles the discovery of valid reasoning paths, particularly for challenging questions requiring extensive deduction. To address this issue, we propose Compress responses for Easy questions and Explore Hard ones (CEEH), a difficulty-aware approach to RL-based efficient reasoning. CEEH dynamically assesses instance difficulty to apply selective entropy regularization: it preserves a diverse search space for currently hard questions to ensure robustness, while permitting aggressive compression on easier instances where the reasoning path is well-established. In addition, we introduce a dynamic optimal-length penalty anchored to the historically shortest correct response, which effectively counteracts entropy-induced length inflation and stabilizes the reward signal. Across six reasoning benchmarks, CEEH consistently reduces response length while maintaining accuracy comparable to the base model, and improves Pass@k relative to length-only optimization.

Figures (6)

• Figure 1: Accuracy–length trade-off in reasoning compression: shorter responses can come at the cost of accuracy and reduced policy entropy.
• Figure 2: The pipeline of our method. The model accuracy is evaluated via GRPO rollouts, and the optimal length is obtained from correct responses. Length penalties are applied only to correct responses when current accuracy exceeds historical accuracy, while entropy regularization is used for questions whose accuracy falls below the average to encourage exploration.
• Figure 3: Training dynamics of policy entropy for R1-Distill-Qwen2.5-7B under different length penalty coefficients. (Left: Maximum-entropy loss (CEEH-ME); Right: Entropy-based advantage (CEEH-EA)).
• Figure 4: Training accuracy on the same dataset, with R1-Distill-Qwen2.5-7B as the base model.
• Figure 5: Distribution of response token counts on AMC23, with R1-Distill-Qwen2.5-7B as the base model.