Revisiting Entropy in Reinforcement Learning for Large Reasoning Models

TL;DR

This paper addresses entropy collapse in RLVR-trained LLMs by conducting large-scale empirical studies of entropy dynamics and its relation to response diversity, calibration, and performance. It identifies three key factors governing entropy: off-policy updates, training data diversity, and clipping thresholds, and provides both theoretical and empirical evidence that tokens with positive advantages drive entropy collapse. The authors propose Progressive Advantage Reweighting to regulate entropy by adjusting loss weights on tokens with positive versus non-positive advantages, demonstrating improved performance while mitigating collapse. The work offers practical guidance for stabilizing RLVR training and enhancing reasoning capabilities in large language models, with implications for data selection, training schedules, and objective design.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a predominant approach for enhancing the reasoning capabilities of large language models (LLMs). However, the entropy of LLMs usually collapses during RLVR training, causing premature convergence to suboptimal local minima and hinder further performance improvement. Although various approaches have been proposed to mitigate entropy collapse, a comprehensive study of entropy in RLVR remains lacking. To address this gap, we conduct extensive experiments to investigate the entropy dynamics of LLMs trained with RLVR and analyze how model entropy correlates with response diversity, calibration, and performance across various benchmarks. Our findings reveal that the number of off-policy updates, the diversity of training data, and the clipping thresholds in the optimization objective are critical factors influencing the entropy of LLMs trained with RLVR. Moreover, we theoretically and empirically demonstrate that tokens with positive advantages are the primary contributors to entropy collapse, and that model entropy can be effectively regulated by adjusting the relative loss weights of tokens with positive and negative advantages during training.

Figures (11)

• Figure 1: Evolution of entropy (solid lines) and N-gram diversity (markers) over training steps under different entropy regularization settings.
• Figure 2: The ratio between the entropy of the LLMs on prompts during training and their entropy on the same prompts before training, plotted against training steps. Solid lines represent results on the AIME 2024, while dashed lines correspond to results on IF-Eval.
• Figure 3: Scatter plot of accuracy versus entropy across AIME 2024, AIME 2025, and MATH500.
• Figure 4: Pearson correlation coefficients between the entropy of LLMs on prompts and their corresponding accuracy across different training steps. "Average" indicates the mean Pearson correlation coefficient computed over AIME 2024, AIME 2025, and MATH500.
• Figure 5: Scatter plot illustrating the relationship between LLM entropy during training and Avg@64 scores on LiveCodeBench. The brown dashed line represents the least-squares regression fit to the data points.