Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Hidden Objective Biases of Group-based Reinforcement Learning

Aleksandar Fontana, Marco Simoni, Giulio Rossolini, Andrea Saracino, Paolo Mori

TL;DR

The paper addresses misalignment between surrogate GRPO-style objectives and true policy improvement in group-based reinforcement learning for LLM post-training. It introduces a unified GRPO-L formulation that encompasses multiple existing methods and analyzes three key dynamics: biased token-prefix gradients from non-uniform weighting, reward-scaling invariance of AdamW in the absence of strong regularization, and momentum-driven overshoot when clipping is enforced. The authors derive theoretical insights, including conditions under which reward scaling does not affect updates and how shared-prefix tokens can be biased by weighting schemes, as well as a detailed account of Adam overshoot and its practical implications for multi-step updates. These findings highlight fundamental limitations of current surrogate objectives and offer principled directions for designing more faithful training signals that better reflect the underlying optimization goals.

Abstract

Group-based reinforcement learning methods, like Group Relative Policy Optimization (GRPO), are widely used nowadays to post-train large language models. Despite their empirical success, they exhibit structural mismatches between reward optimization and the underlying training objective. In this paper, we present a theoretical analysis of GRPO style methods by studying them within a unified surrogate formulation. This perspective reveals recurring properties that affect all the methods under analysis: (i) non-uniform group weighting induces systematic gradient biases on shared prefix tokens; (ii) interactions with the AdamW optimizer make training dynamics largely insensitive to reward scaling; and (iii) optimizer momentum can push policy updates beyond the intended clipping region under repeated optimization steps. We believe that these findings highlight fundamental limitations of current approaches and provide principled guidance for the design of future formulations.

On the Hidden Objective Biases of Group-based Reinforcement Learning

TL;DR

The paper addresses misalignment between surrogate GRPO-style objectives and true policy improvement in group-based reinforcement learning for LLM post-training. It introduces a unified GRPO-L formulation that encompasses multiple existing methods and analyzes three key dynamics: biased token-prefix gradients from non-uniform weighting, reward-scaling invariance of AdamW in the absence of strong regularization, and momentum-driven overshoot when clipping is enforced. The authors derive theoretical insights, including conditions under which reward scaling does not affect updates and how shared-prefix tokens can be biased by weighting schemes, as well as a detailed account of Adam overshoot and its practical implications for multi-step updates. These findings highlight fundamental limitations of current surrogate objectives and offer principled directions for designing more faithful training signals that better reflect the underlying optimization goals.

Abstract

Group-based reinforcement learning methods, like Group Relative Policy Optimization (GRPO), are widely used nowadays to post-train large language models. Despite their empirical success, they exhibit structural mismatches between reward optimization and the underlying training objective. In this paper, we present a theoretical analysis of GRPO style methods by studying them within a unified surrogate formulation. This perspective reveals recurring properties that affect all the methods under analysis: (i) non-uniform group weighting induces systematic gradient biases on shared prefix tokens; (ii) interactions with the AdamW optimizer make training dynamics largely insensitive to reward scaling; and (iii) optimizer momentum can push policy updates beyond the intended clipping region under repeated optimization steps. We believe that these findings highlight fundamental limitations of current approaches and provide principled guidance for the design of future formulations.
Paper Structure (27 sections, 7 theorems, 58 equations, 2 figures, 1 table)

This paper contains 27 sections, 7 theorems, 58 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Unified Formulation
  4. Biases in Token-level Gradients
  5. Effects of AdamW Optimizer
  6. Reward Scaling.
  7. Adam Overshoot.
  8. Conclusion
  9. Inadequacy of the Surrogate Loss as a Performance Proxy
  10. General form of the objective.
  11. Scenario A: First optimization step ($\rho_{i,t}(\theta) = 1$)
  12. Conclusion of Scenario A.
  13. Scenario B: Multiple optimization steps ($\rho_{i,t}(\theta) \neq 1$)
  14. Derivation of the Gradient for $\mathcal{J}_{\text{GRPO-L}}(\theta)$
  15. Gradient of the GRPO style objective
  16. ...and 12 more sections

Key Result

Proposition 1

Consider a policy $\pi_\theta$ optimized with Eq. eq:grpol via centered advantages (Eq. eq:advantage). For any subset of answers $\tilde{G} \subseteq G$ sharing a common prefix $y_{i, 1:|k|}$, the gradient with respect to this prefix is modulated by the aggregate term $\mathcal{W}_{\text{agg}} = \su

Theorems & Definitions (10)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Proposition 5: Advantage scaling
  • proof
  • Proposition 6: Moment scaling
  • proof
  • Proposition 7: Momentum overshoot
  • proof