Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

TL;DR

An explanation through the lens of local curvature of the sequence-level policy gradient of when standard deviation normalization implements an adaptive gradient is provided of when std normalization helps in Group Relative Policy Optimization.

Abstract

Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we provide an explanation through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, under mild conditions, GRPO enjoys a strictly improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. Empirically, our analysis on GSM8K and MATH benchmarks reveals three distinct training phases governed by the interplay between feature orthogonality and reward variance: (I) an early acceleration phase where high variance and orthogonality favor adaptive scaling; (II) a relatively stable transition phase; and (III) a late-stage regime where the loss of orthogonality limits further gains. Together, these results provide a principled account of when std normalization helps in GRPO, and offer broader insights into the design of critic-free RL algorithms.

Figures (5)

• Figure 1: Empirical validation of Assumption \ref{['ass:ass_2']}.
• Figure 2: Empirical validation of Assumption \ref{['ass:ass_4']}
• Figure 3: Training accuracy vs. iterations on GSM8K Easy (a) and Hard (b) with phase annotations. standard (green) uses variance normalization; no_std (orange) omits normalization. Shaded regions indicate three training regimes based on gradient geometry: Phase I (blue, 0--100): near-orthogonal regime where high reward variance leads to comparable performance despite favorable geometric conditions; Phase II (green, 100--300): low-variance regime where reduced orthogonality is offset by stable cross-prompt interactions, enabling standard to establish its advantage; Phase III (red, 300--500): high-variance regime where increased heterogeneity in gradient similarities limits further acceleration, though standard maintains its 2--3% lead. Curves are smoothed with exponential moving average ($\alpha = 0.18$).
• Figure 4: Evolution of Pairwise Gradient Cosine Similarity During GRPO Training on GSM8K. Distribution of cosine similarities between per-prompt gradient pairs across 100 questions ( 4,950 pairs) at three training stages. The red dashed line indicates $\cos =0$ (orthogonality), and the orange solid line indicates the mean. (Left) Step 0: Near-perfect orthogonality with over 90% of pairs satisfying $|\cos |<0.1$. (Middle) Step 260: Gradients begin developing positive correlations while maintaining low variance (std $=0.066$ ). (Right) Step 500: Increased variance (std $=0.130$ ) with the distribution spreading while remaining centered near zero.
• Figure 5: Training dynamics on MATH Level 2 subset. Similar to the GSM8K experiments, Normalized GRPO (standard) demonstrates faster initial convergence and maintains a lead for the majority of training. The narrowing gap in the final steps further evidences the impact of cross-prompt interference in the late training stages.

Theorems & Definitions (12)

• Remark 2.1
• Lemma 3.1
• Lemma 3.2
• Lemma 3.3: Non-uniform local smoothness
• Theorem 3.1: Convergence rate of REINFORCE
• Theorem 3.2: Convergence rate of GRPO
• proof : Proof sketch
• Theorem 3.3
• Theorem 3.4
• proof : Proof sketch