Your Group-Relative Advantage Is Biased

TL;DR

Group-relative RLVR exhibits a systematic bias in the estimated advantage: hard prompts tend to be undervalued and easy prompts overvalued due to the empirical group baseline. The authors formalize this bias, prove its dependence on prompt difficulty, and introduce History-Aware Adaptive Difficulty Weighting (HA-DW), which uses an evolving difficulty anchor and history signals to reweight advantages. Theoretical results show HA-DW can reduce bias in expectation, and extensive experiments on five mathematical reasoning benchmarks across multiple model families demonstrate consistent performance gains with GRPO and related algorithms. This work highlights a crucial bias in current RLVR practice and offers a practical, plug-in method to enhance robustness and sample efficiency for reasoning-focused LLMs.

Abstract

Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.

Figures (8)

• Figure 1: (a) Comparison of the performance of RL algorithms with and without HA-DW on Qwen3-4B-Base across five mathematical reasoning benchmarks. (b) Significant biased advantage estimation on the MATH dataset under 8 and 128 rollouts. (c) Performance gain by GRPO+HA-DW on MATH500 stratified by difficulty levels.
• Figure 2: Illustration of advantage estimation bias as a function of $p_t$ and group size $G$.
• Figure 3: HA-DW consists of two collaborative phases. In the first phase, an evolving difficulty anchor incorporates cross-batch historical information by propagating the model’s prior through a history buffer, capturing long-term reward trends. In the second phase, prompt weights are adaptively adjusted based on their estimated difficulty under the model’s evolving state, compensating for biased advantage estimates.
• Figure 4: Comparison of training dynamics under different training strategies. Average accuracy across five benchmarks, training reward and response length of Qwen3-4B-Base and Qwen3-8B-Base on different training methods.
• Figure 5: Illustration of advantage bias under truncated Gaussian rewards for different group sizes.

Theorems & Definitions (22)

• Definition 1: Expected Reward
• Definition 2: Expected Advantage
• Definition 3: Prompt Difficulty
• Theorem 1
• Theorem 2
• Corollary 1
• Corollary 2
• Corollary 3
• Lemma 1: Baseline Rectification
• Theorem 3