Rediscovering Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning

TL;DR

The paper identifies fixed entropy regularization as a bottleneck for RLVR in LLMs due to entropy collapse or explosion and proposes Adaptive Entropy Regularization (AER). AER jointly uses (i) difficulty-aware per-sample coefficient allocation, (ii) an initial-anchored target entropy, and (iii) a dynamic global coefficient to keep policy entropy near a target $H^{\star}$ throughout training. Empirical results on mathematical reasoning benchmarks across model scales show that AER improves both reasoning accuracy (pass@1) and exploration (pass@k), with clear ablations validating the contribution of each component. The work demonstrates that adaptive entropy control can unlock the potential of entropy regularization for RLVR in LLMs, enabling more reliable and diverse reasoning trajectories.

Abstract

Reasoning ability has become a defining capability of Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) emerging as a key paradigm to enhance it. However, RLVR training often suffers from policy entropy collapse, where the policy becomes overly deterministic, hindering exploration and limiting reasoning performance. While entropy regularization is a common remedy, its effectiveness is highly sensitive to the fixed coefficient, making it unstable across tasks and models. In this work, we revisit entropy regularization in RLVR and argue that its potential has been largely underestimated. Our analysis shows that (i) tasks of varying difficulty demand distinct exploration intensities, and (ii) balanced exploration may require the policy entropy to be maintained within a moderate range below its initial level. Therefore, we propose Adaptive Entropy Regularization (AER)--a framework that dynamically balances exploration and exploitation via three components: difficulty-aware coefficient allocation, initial-anchored target entropy, and dynamic global coefficient adjustment. Experiments on multiple mathematical reasoning benchmarks show that AER consistently outperforms baselines, improving both reasoning accuracy and exploration capability.

Figures (4)

• Figure 1: An overview of the AER framework.
• Figure 2: Preliminary experimental results. (a-b) we show the effect of different entropy coefficients on test accuracy and average token length on easy and difficult datasets, respectively. (c) we demonstrate that different base models, datasets, and sampling temperatures will significantly affect the value of the initial entropy.
• Figure 3: Pass@$k$ and training dynamics (2×3 grid). Left column: pass@$k$ on AIME24 (a) and AIME25 (d) as $k$ scales. Right two columns: training dynamics—reward (b), policy entropy (c), response length (e)—and test score on AIME25 (f) over steps.
• Figure 4: AER ablations and generalization. (a) Pivot accuracy $\rho$ ablation (C1, $\tau{=}0.4$). (b) Reduction ratio $\tau$ ablation (C2, $\rho{=}0.2$). (c) Generalization on dapo_math_17k: Qwen3-8B-Base test score. (d) Generalization on dapo_math_17k: Qwen2.5-7B-Base test score.