The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View

TL;DR

The paper examines whether reinforcement learning with verifiable rewards (RLVR) expands or shrinks the reasoning capabilities of large language models. It introduces a two-stage probability mass dynamic—exploitation followed by exploration—to explain how initial training can narrow capabilities while extended training can enable novel reasoning, especially when updates emphasize relative negative gradients (-N). The authors provide theoretical derivations for logit updates under policy-gradient methods and validate them with toy and real-world experiments, showing that GRPO-N and GSPO-N can maintain diversity and improve reasoning performance. The work offers a practical direction for prolonging RLVR training to unlock more advanced capabilities and frames questions for efficient probability-mass allocation in future research.

Abstract

The ongoing debate on whether reinforcement learning with verifiable rewards (RLVR) expands or shrinks the reasoning capabilities of large language models (LLMs) remains unresolved. Some studies contend that RLVR mainly improves sampling efficiency but at the expense of diversity and exploratory capacity, resulting in capability boundary shrinkage. In contrast, others demonstrate that prolonged training can lead to the emergence of novel reasoning strategies, suggesting capability boundary expansion. To reconcile these contradictory findings, we theoretically and empirically show that both perspectives are partially valid-each aligning with a separate phase in an inherent two-stage probability mass dynamic: (1) Exploitation stage: initially, the model primarily samples explored high-reward and low-reward tokens, while rarely selecting the potentially optimal token. Positive advantage estimates increase the probability of high-reward tokens and decrease those of low-reward tokens, yet the optimal token's probability remains largely unchanged during this stage. (2) Exploration stage: as training advances, the growth rate of previously acquired high-reward tokens slows as their probabilities approach saturation. When a potentially optimal token-now receiving positive advantage estimates-is occasionally sampled, its probability increases, while those of the originally high-reward tokens decrease. This dynamic suggests that over-exploitation during the exploitation stage may lead to capability boundary shrinkage, whereas prolonged training into the exploration stage can promote an expansion of the reasoning capability boundary. Building upon our insights, we revisit the potential of only using relative negative gradients for prolonging training, providing a theoretical and empirical foundation for the development of more advanced reasoning capabilities.

Figures (7)

• Figure 1: The probability mass dynamics of policy optimization across varying action rewards $r$ and initial policy probabilities $\pi$. Each sub-figure corresponds only to the indicated rewards and probabilities. The first row compares the impact of different initial policy probabilities under identical rewards, while the second row compares the effect of varying rewards given the same initial policy.
• Figure 2: Comparison of the training dynamics of GRPO, GRPO-N, GSPO, and GSPO-N on the MATH benchmark across training steps, using the Qwen2.5-Math-7B model with a prompt batch size of 1,024. Left Part: (Left) the greedy decoding accuracy on the MATH test set and (Center Left) the model's entropy on the MATH test set. Right Part: (Center Right) the actor entropy loss and (Right) critic rewards mean during training. GRPO causes the entropy of the base model to collapse over the course of training, suggesting a loss of exploratory capability. In contrast, GRPO-N, GSPO, and GSPO-N all exhibit a pattern where entropy initially decreases and then increases. Notably, the entropy of GRPO-N significantly surpasses that of the base model. All algorithms achieve competitive performance in both greedy decoding accuracy and critic rewards mean.
• Figure 3: A comparison of the correct responses of GRPO and GRPO-N (a test case from AMC 2023). The key reasoning steps are presented here, see Appendix \ref{['app:case']} for full procedure.
• Figure 4: Comparison of the training dynamics of GRPO, GRPO-N on the MATH benchmark across training steps, using the LLama-3.2-3B-Instruct model with a prompt batch size of 1,024. Left Part: (Left) the greedy decoding accuracy on the MATH test set and (Center Left) the model's entropy on the MATH test set. Right Part: (Center Right) the actor entropy loss and (Right) critic rewards mean during training.
• Figure 5: Dynamics of the policy probability mass during optimization for different numbers of rollout samples ($[2, 3, 5, 10]$), with action rewards $r$ and initial policy probabilities $\pi$ held constant.

Theorems & Definitions (11)

• Remark 1
• Lemma 1: Logits Update for Softmax Parameterization
• Remark 2: Bidirectional Update Rule
• Theorem 1: The Expected Logits Update
• Remark 3: A Two-Stage Dynamic of Exploitation and Exploration
• Remark 4
• proof
• proof
• Proposition 1
• proof