Entropy Ratio Clipping as a Soft Global Constraint for Stable Reinforcement Learning

TL;DR

The paper tackles training instability in off-policy reinforcement learning for large language model fine-tuning by addressing trust-region drift not captured by local PPO-clip. It introduces the entropy ratio as a global measure of policy exploration change and proposes Entropy Ratio Clipping (ERC), a bidirectional constraint that clamps updates when the entropy ratio deviates beyond set bounds. ERC is integrated with DAPO and GPPO, yielding consistent performance gains and smoother training dynamics across multiple math-reasoning benchmarks and model scales. The work demonstrates that global distribution-level constraints can enhance stability without sacrificing exploration, with potential applicability beyond the evaluated domains.

Abstract

Large language model post-training relies on reinforcement learning to improve model capability and alignment quality. However, the off-policy training paradigm introduces distribution shift, which often pushes the policy beyond the trust region, leading to training instabilities manifested as fluctuations in policy entropy and unstable gradients. Although PPO-Clip mitigates this issue through importance clipping, it still overlooks the global distributional shift of actions. To address these challenges, we propose using the entropy ratio between the current and previous policies as a new global metric that effectively quantifies the relative change in policy exploration throughout updates. Building on this metric, we introduce an \textbf{Entropy Ratio Clipping} (ERC) mechanism that imposes bidirectional constraints on the entropy ratio. This stabilizes policy updates at the global distribution level and compensates for the inability of PPO-clip to regulate probability shifts of un-sampled actions. We integrate ERC into both DAPO and GPPO reinforcement learning algorithms. Experiments across multiple benchmarks show that ERC consistently improves performance.

Figures (8)

• Figure 1: (a): Scatter plot showing the relationship between token-wise sampling probability and entropy ratio during RL training. (b): Comparison of the optimization objectives for DAPO and DAPO augmented with ERC. ERC extends the standard PPO-clip objective in DAPO by introducing an additional clipping term on the entropy ratio $\rho_{i,t}$, thereby enforcing a global distribution-level constraint. (c): Comparison of the trust regions with and without ERC. By applying bidirectional clipping on the entropy ratio, ERC further tightens the trust region beyond PPO-clip, effectively mitigating trust-region drift.
• Figure 2: Training dynamics of entropy, gradient norm and benchmark accuracy on DeepSeek-R1-Distill-Qwen-7B, comparing various baseline method with and without the proposed ERC mechanism.
• Figure 3: Visualization of the clipping regions. Red points indicate tokens clipped for exceeding the upper bound of the entropy ratio, while yellow points indicate tokens clipped for falling below the lower bound. Blue points represent tokens that were not clipped. The entropy ratio clipping shown here is applied on top of the standard importance ratio clipping.
• Figure 4: Scatter plot illustrating the relationship between sampled token probabilities and the entropy of their corresponding distributions. Blue points represent tokens that are not clipped by the ERC mechanism, while orange points denote tokens that are clipped by the entropy ratio constraint.
• Figure 5: Word cloud visualization of tokens unclipped by and clipped by the ERC mechanism.