Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification

Youssef Mroueh

TL;DR

This work analyzes reinforcement learning with verifiable binary rewards for LLMs under GRPO, revealing that reward whitening induces an adaptive weighted contrastive loss and enabling a tractable fixed-point analysis of policy updates. It develops and contrasts multiple GRPO variants—mean–variance vs mean-only normalization, reference vs mirror KL regularization, and clipping vs no-clipping—with explicit closed-form PoS recursions and fixed points, showing that GRPO can amplify the probability of success relative to a reference. Theoretical results establish conditions for fixed-point convergence, monotone improvement (notably for Mirror GRPO), and equivalence between mean–variance and mean-only schemes via adaptive KL strength. Empirical results on GSM8K demonstrate substantial improvements in success rates, validating the theory and offering practical guidance for deploying verifiable rewards to enhance LLM reasoning tasks.

Abstract

Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model $π_{\mathrm{ref}}$, or combining both forms of regularization. For each, the optimal policy $π_n$ admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under $π_{n-1}$, as well as the policies $π_{n-1}$ and $π_{\mathrm{ref}}$. Iterating results in a sequence $\{π_n\}$ whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.

Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification

TL;DR

This work analyzes reinforcement learning with verifiable binary rewards for LLMs under GRPO, revealing that reward whitening induces an adaptive weighted contrastive loss and enabling a tractable fixed-point analysis of policy updates. It develops and contrasts multiple GRPO variants—mean–variance vs mean-only normalization, reference vs mirror KL regularization, and clipping vs no-clipping—with explicit closed-form PoS recursions and fixed points, showing that GRPO can amplify the probability of success relative to a reference. Theoretical results establish conditions for fixed-point convergence, monotone improvement (notably for Mirror GRPO), and equivalence between mean–variance and mean-only schemes via adaptive KL strength. Empirical results on GSM8K demonstrate substantial improvements in success rates, validating the theory and offering practical guidance for deploying verifiable rewards to enhance LLM reasoning tasks.

Abstract

Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model , or combining both forms of regularization. For each, the optimal policy admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under , as well as the policies and . Iterating results in a sequence whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.

Paper Structure

This paper contains 27 sections, 14 theorems, 81 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. GRPO With verifiable Rewards as an Adaptive Weighted Contrastive Loss
  3. Whitening the Rewards in GRPO As means of Calibration
  4. Calibrated reward behavior.
  5. GRPO with verifiable Reward As a Weighted Contrastive Loss
  6. Stabilized GRPO
  7. Stabilized GRPO with No Clipping
  8. GRPO Iterations
  9. GRPO Dynamics: Fixed Point iteration for Probability of Success
  10. GRPO: Fixed Point Iteration and Success Amplification
  11. Mirror GRPO: Mirror Descent with GRPO Calibrated Reward
  12. Dr. GRPO and mean-only Normalization
  13. Discussion and Conclusion
  14. Experimental Validation
  15. Algorithm
  16. ...and 12 more sections

Key Result

Theorem 1

Optimal GRPO iterations policies solving eq:GRPO_iterations satisfy the following recursion, for $n\geq 1$: where $Z_{n-1}(q) = p_{\mathrm{ref}}(q) \exp \left(\frac{1}{\beta} \omega^+_{\varepsilon}(p_{n-1}(q)) \right) + (1-p_{\mathrm{ref}}(q)) \exp \left(- \frac{1}{\beta} \omega^-_{\varepsilon} (p_{n-1} (q)) \right) ,$ where the weights $\omega^+_\varepsilon$ and $\omega^-_{\varepsilon}$ are

Figures (5)

  • Figure 1: Weighting of GRPO with the probability of success of the old policy.
  • Figure 2: Fixed points as function of $\beta$ and $p_{\mathrm{ref}}$ for $\varepsilon = 1e^{-5}$.
  • Figure 3: Success rate trajectory of the model on randomly selected prompts along the GRPO iters.
  • Figure 4: Lower bound on $\beta$ to ensure local convergence of GRPO fixed point iteration.
  • Figure 5: GRPO Recursion and convergence to fixed points of $h_{\varepsilon}$, for $\varepsilon = 1e^{-5}$

Theorems & Definitions (28)

  • Theorem 1: GRPO Policy Dynamics
  • Theorem 2: GRPO's Probability of Success Fixed Point Iteration
  • Remark 1: Importance of $\varepsilon >0$
  • Proposition 1: Properties of $h_{\varepsilon, p_{\mathrm{ref}}}$
  • Theorem 3: GRPO amplifies the probability of success
  • Theorem 4: Mirror-GRPO, $\alpha=0$
  • Theorem 5: Monotone Improvement and Absorbing Fixed Points
  • proof : Proof of Theorem \ref{['theo:piexpression']}
  • proof : Proof of Theorem \ref{['theo:IterationProba']}
  • proof : Proof of Proposition \ref{['pro:propreties']}
  • ...and 18 more