Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropy-Preserving Reinforcement Learning

Aleksei Petrenko, Ben Lipkin, Kevin Chen, Erik Wijmans, Marco Cusumano-Towner, Raja Giryes, Philipp Krähenbühl

Abstract

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.

Entropy-Preserving Reinforcement Learning

Abstract

Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.
Paper Structure (51 sections, 18 theorems, 56 equations, 10 figures, 5 tables)

This paper contains 51 sections, 18 theorems, 56 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Theory: Entropy dynamics of policy gradient
  4. Entropy dynamics of PPO.
  5. Entropy dynamics of DAPO.
  6. Entropy dynamics of GSPO.
  7. Summary.
  8. Empirical findings: Implementation Details Affecting Entropy
  9. 16-bit Quantization of Model Outputs Affects Clipping
  10. Float16 vs BFloat16 training
  11. Explicit Entropy Control Methods
  12. REPO: Regulated entropy policy optimization
  13. REPO-D (Decorrelate).
  14. REPO-R (Rescale).
  15. Adaptive controller.
  16. ...and 36 more sections

Key Result

Theorem 1

Given a policy gradient update $\widehat{{\color{black}\theta}} := {\color{black}\theta} + \alpha\cdot {\color{black}\nabla}_{\color{black}\theta} {\color{black}\mathcal{J}}_\mathrm{MDP}({\color{black}\boldsymbol{s}})$, the expected change in entropy is approximately: ${\color{black}L}({\color{black}\boldsymbol{s}},{\color{black}a})\mathrel{\overset{\raisebox{-0.25ex}{\tiny def}}{=}}\log{\color{b

Figures (10)

  • Figure 1: Top: Evolution of the average per-token entropy and test accuracy during training for several baselines (GRPO, LOOP, DAPO, GSPO) and their entropy regularized versions (REPO). Each curve shows the average trajectory over several training runs with different seeds. Bottom: Cumulative entropy experienced during training up to a given checkpoint is positively correlated with the test accuracy. Each point is a checkpoint of a single training run (best-performing checkpoint per run highlighted). Algorithms that collapse the entropy early (see e.g. Qwen-3-8B on AppWorld; middle column) perform significantly worse than algorithms that maintain a steady entropy during training.
  • Figure 2: (a) Fraction of tokens hitting clip bounds with 16-bit rounding: ${\epsilon}_{\text{high}}$ is reached more often and ${\epsilon}_{\text{low}}$ less often, hindering the promotion of low-probability actions. (b) Average probability difference and max importance weight ratio between vLLM inference and training forward pass.
  • Figure 3: FP16 training on Qwen-3-8B AppWorld and clipping fix lead to a qualitative change: DAPO entropy collapse transitions to entropy increase.
  • Figure 4: Entropy-preserving methods compared to baselines with Qwen-3-8B on AppWorld.
  • Figure 5: Entropy-preserving RL training with Qwen-3-32B on AppWorld.
  • ...and 5 more figures

Theorems & Definitions (32)

  • Theorem 1
  • Corollary 1
  • Theorem 2
  • Theorem 3
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • ...and 22 more