Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

TL;DR

This work identifies an implicit advantage symmetry in Group Relative Advantage Estimation (GRAE) within GRPO that bottlenecks exploration and difficulty adaptation in reinforcement learning with verifiable rewards. By formalizing the RLVR framework and analyzing symmetry at group and sample levels, the authors show that standard GRPO underutilizes unsampled trajectories and overemphasizes medium-difficulty samples, prompting suboptimal learning. They propose Asymmetric GRAE (A-GRAE), which introduces asymmetric exploration and a curriculum-like sample weighting that shifts from easier to harder tasks as training progresses, guided by batch mean rewards. Across seven benchmarks spanning text and vision-language reasoning, A-GRAE consistently improves GRPO and its variants on both LLMs and MLLMs, demonstrating enhanced reasoning accuracy and solution diversity with improved training stability.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we argue that these bottlenecks stem from an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE). This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically suppressing the advantages of correct trajectories encourages essential exploration. (ii) learning efficiency is maximized by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs.

Figures (10)

• Figure 1: The two-fold implicit advantage symmetry problem of GRAE in GRPO. At the group level, the advantage weights for correct trajectories equal those of incorrect trajectories. This symmetry leads to the logits of low-probability correct paths unchanged within the behavior space, thereby hindering the model's exploration. At the sample level, samples of medium-difficulty exhibit the largest sum of absolute advantage values, which leads to insufficient training on harder data.
• Figure 2: Experimental results on breaking group-level symmetry using Qwen2.5-Math-7B. We amplify (Positive-Dominant) or suppress (Negative-Dominant) the advantages of correct trajectories to compare their performance with that of GRPO and the base model.The performance is evaluated using Pass@$k$ ($k=\{1,2,4,8,16,32,64,128,256\}$).
• Figure 3: Entropy dynamics across the three groups in Experiment I on the training set. Notably, the Negative-Dominant group exhibits a monotonic increase in entropy except at the very beginning, while the other groups show the opposite behavior.
• Figure 4: Experimental results on breaking sample-level symmetry using Qwen2.5-Math-7B. We rescaling the advantages to shift the learning focus toward harder queries (Hard-Focused) or easier queries (Easy-Focused) to compare their performance with that of GRPO and the base model.The performance is evaluated using Pass@$k$ ($k=\{1,2,4,8,16,32,64,128,256\}$).
• Figure 5: The within-batch count of correct sampling responses on the training set. Easy-Focused exhibits the most rapid initial convergence during the early stages of training, whereas Hard-Focused maintains a sustained upward trajectory in the later phases, eventually achieving superior performance.