Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic

TL;DR

A unified framework to understand GRPO through the lens of classical U-statistics is provided, demonstrating that the GRPO policy gradient is inherently a U-statistic, allowing it to characterize its mean squared error (MSE), derive the finite-sample error bound and asymptotic distribution of the suboptimality gap for its learned policy.

Abstract

Group relative policy optimization (GRPO), a core methodological component of DeepSeekMath and DeepSeek-R1, has emerged as a cornerstone for scaling reasoning capabilities of large language models. Despite its widespread adoption and the proliferation of follow-up works, the theoretical properties of GRPO remain less studied. This paper provides a unified framework to understand GRPO through the lens of classical U-statistics. We demonstrate that the GRPO policy gradient is inherently a U-statistic, allowing us to characterize its mean squared error (MSE), derive the finite-sample error bound and asymptotic distribution of the suboptimality gap for its learned policy. Our findings reveal that GRPO is asymptotically equivalent to an oracle policy gradient algorithm -- one with access to a value function that quantifies the goodness of its learning policy at each training iteration -- and achieves asymptotically optimal performance within a broad class of policy gradient algorithms. Furthermore, we establish a universal scaling law that offers principled guidance for selecting the optimal group size. Empirical experiments further validate our theoretical findings, demonstrating that the optimal group size is universal, and verify the oracle property of GRPO.

Figures (5)

• Figure 1: An illustration of LLM reasoning. The example shows a prompt and the corresponding model output, consisting of a reasoning trace and a final solution.
• Figure 2: Overview of the GRPO pipeline. For each prompt, GRPO samples multiple reasoning traces to generate outputs, which are then evaluated by a reward model to measure their quality. These rewards are compared against the group mean and standardized to compute the advantage function, which is used to update the policy model, subject to KL regularization with respect to a reference model.
• Figure 3: Roadmap of our theoretical results.
• Figure 4: MSEs of three policy gradient estimators (vanilla, GRPO-type, and oracle) under three model configurations (base, instruct and in-context learning (ICL)) for different group sizes. Error bars represent 95% confidence intervals of the empirically estimated MSEs.
• Figure 5: Test accuracy of GRPO-fine-tuned models at different training steps with a fixed sampling budget of $N=B \times G = 1024$ per prompt. Both training and evaluation are conducted on GSM8K. Each curve shows accuracy as a function of the group size $G \in \{4, 8, 16, 32, 64, 128\}$. Results are averaged over five independent runs, with shaded regions visualizing 95% confidence bands.

Theorems & Definitions (10)

• Lemma 1: Gradient estimator as a U-statistic
• Theorem 2: MSE conditional on the prompt
• Proposition 3: MSE in the minibatch setting
• Corollary 4: Oracle property of gradient estimator
• Corollary 5: Optimality of gradient estimator
• Lemma 6: Finite-sample sub-optimality gap
• Theorem 7: Scaling law for GRPO
• Theorem 8: Consistency & asymptotic distribution
• Corollary 9: Oracle property of the policy
• Corollary 10: Optimality of the policy