Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization

Xuan Xie, Xuan Wang, Wenjie Wang, Shuai Chen, Wei Lin

TL;DR

DaGRPO tackles gradient instability in Group Relative Policy Optimization by introducing distinctiveness-aware mechanisms: sequence-level gradient rectification to filter low-distinctiveness samples and off-policy data augmentation to provide high-quality reasoning anchors. The approach stabilizes training, accelerates emergence of long-chain reasoning, and yields state-of-the-art performance on 9 math and OOD benchmarks, including a +4.7% average gain over GRPO on in-distribution math tasks. Ablations show the gradient-rectification component boosts performance even without off-policy data, while anchors significantly aid hard tasks and generalization. Limitations include the computational cost of LLM-based scoring and dependence on external expert data for anchors, pointing to directions for future efficiency improvements and self-bootstrapping methods.

Abstract

The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift from superficial instruction following to rigorous long-horizon reasoning. While Group Relative Policy Optimization (GRPO) has emerged as a pivotal mechanism for eliciting such post-training reasoning capabilities due to its exceptional performance, it remains plagued by significant training instability and poor sample efficiency. We theoretically identify the root cause of these issues as the lack of distinctiveness within on-policy rollouts: for routine queries, highly homogeneous samples induce destructive gradient conflicts; whereas for hard queries, the scarcity of valid positive samples results in ineffective optimization. To bridge this gap, we propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO). DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness, thereby eradicating gradient conflicts at the source; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks. Extensive experiments across 9 mathematical reasoning and out-of-distribution (OOD) generalization benchmarks demonstrate that DaGRPO significantly surpasses existing SFT, GRPO, and hybrid baselines, achieving new state-of-the-art performance (e.g., a +4.7% average accuracy gain on math benchmarks). Furthermore, in-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.

DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization

TL;DR

DaGRPO tackles gradient instability in Group Relative Policy Optimization by introducing distinctiveness-aware mechanisms: sequence-level gradient rectification to filter low-distinctiveness samples and off-policy data augmentation to provide high-quality reasoning anchors. The approach stabilizes training, accelerates emergence of long-chain reasoning, and yields state-of-the-art performance on 9 math and OOD benchmarks, including a +4.7% average gain over GRPO on in-distribution math tasks. Ablations show the gradient-rectification component boosts performance even without off-policy data, while anchors significantly aid hard tasks and generalization. Limitations include the computational cost of LLM-based scoring and dependence on external expert data for anchors, pointing to directions for future efficiency improvements and self-bootstrapping methods.

Abstract

The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift from superficial instruction following to rigorous long-horizon reasoning. While Group Relative Policy Optimization (GRPO) has emerged as a pivotal mechanism for eliciting such post-training reasoning capabilities due to its exceptional performance, it remains plagued by significant training instability and poor sample efficiency. We theoretically identify the root cause of these issues as the lack of distinctiveness within on-policy rollouts: for routine queries, highly homogeneous samples induce destructive gradient conflicts; whereas for hard queries, the scarcity of valid positive samples results in ineffective optimization. To bridge this gap, we propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO). DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness, thereby eradicating gradient conflicts at the source; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks. Extensive experiments across 9 mathematical reasoning and out-of-distribution (OOD) generalization benchmarks demonstrate that DaGRPO significantly surpasses existing SFT, GRPO, and hybrid baselines, achieving new state-of-the-art performance (e.g., a +4.7% average accuracy gain on math benchmarks). Furthermore, in-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.

Paper Structure

This paper contains 14 sections, 11 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Analysis of Gradient Conflict in GRPO
  5. Method
  6. Experimental Setups
  7. Training Datasets
  8. Benchmarks
  9. Baselines
  10. Implementation Details
  11. Experimental Results
  12. Main Results
  13. Training Dynamics Analysis
  14. Conclusion

Figures (3)

  • Figure 1: Visualization of Gradient Conflict. Using Qwen2.5-0.5B-Instruct with group size $G=8$, we visualize the gradient directions of distinct responses for an identical prompt. The color spectrum (Red=High Score, Blue=Low Score) reflects GPT-4o evaluations.
  • Figure 2: Training Dynamics. Comparison of training curves between DaGRPO and baseline GRPO over 500 steps. Left: Average outcome rewards per step. Middle: Average response length (tokens). Right: Policy entropy.
  • Figure 3: Evolution of Gradient Norms throughout the training process. The y-axis is plotted on a logarithmic scale.