Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management

Xuerui Su, Liya Guo, Yue Wang, Yi Zhu, Zhiming Ma, Zun Wang, Yuting Liu

TL;DR

DGRO tackles the exploration–exploitation trade-off in LLM reasoning by decoupling the regularization effect into two coefficients, $\beta_1$ and $\beta_2$, and by using the group mean reward $\bar{r}(x)$ as a tractable surrogate for the soft value $V(x,\beta)$. The paper provides theoretical results bounding the approximation error in different regimes of $\beta$ and analyzes how reward variance shapes gradient dynamics, linking variance to convergence speed. Empirically, DGRO with rule-based rewards achieves state-of-the-art performance on Logic benchmarks and shows strong generalization on mathematics reasoning tasks, outperforming several baselines under online updates. The work unifies several reward-optimization strategies, including DRO and Kimi, and offers practical guidance for designing exploration–exploitation controls and variance management in RL for reasoning.

Abstract

Inference scaling further accelerates Large Language Models (LLMs) toward Artificial General Intelligence (AGI), with large-scale Reinforcement Learning (RL) to unleash long Chain-of-Thought reasoning. Most contemporary reasoning approaches usually rely on handcrafted rule-based reward functions. However, the tarde-offs of exploration and exploitation in RL algorithms involves multiple complex considerations, and the theoretical and empirical impacts of manually designed reward functions remain insufficiently explored. In this paper, we propose Decoupled Group Reward Optimization (DGRO), a general RL algorithm for LLM reasoning. On the one hand, DGRO decouples the traditional regularization coefficient into two independent hyperparameters: one scales the policy gradient term, and the other regulates the distance from the sampling policy. This decoupling not only enables precise control over balancing exploration and exploitation, but also can be seamlessly extended to Online Policy Mirror Descent (OPMD) algorithms in Kimi k1.5 and Direct Reward Optimization. On the other hand, we observe that reward variance significantly affects both convergence speed and final model performance. We conduct both theoretical analysis and extensive empirical validation to assess DGRO, including a detailed ablation study that investigates its performance and optimization dynamics. Experimental results show that DGRO achieves state-of-the-art performance on the Logic dataset with an average accuracy of 96.9\%, and demonstrates strong generalization across mathematical benchmarks.

DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management

TL;DR

DGRO tackles the exploration–exploitation trade-off in LLM reasoning by decoupling the regularization effect into two coefficients, and , and by using the group mean reward as a tractable surrogate for the soft value . The paper provides theoretical results bounding the approximation error in different regimes of and analyzes how reward variance shapes gradient dynamics, linking variance to convergence speed. Empirically, DGRO with rule-based rewards achieves state-of-the-art performance on Logic benchmarks and shows strong generalization on mathematics reasoning tasks, outperforming several baselines under online updates. The work unifies several reward-optimization strategies, including DRO and Kimi, and offers practical guidance for designing exploration–exploitation controls and variance management in RL for reasoning.

Abstract

Inference scaling further accelerates Large Language Models (LLMs) toward Artificial General Intelligence (AGI), with large-scale Reinforcement Learning (RL) to unleash long Chain-of-Thought reasoning. Most contemporary reasoning approaches usually rely on handcrafted rule-based reward functions. However, the tarde-offs of exploration and exploitation in RL algorithms involves multiple complex considerations, and the theoretical and empirical impacts of manually designed reward functions remain insufficiently explored. In this paper, we propose Decoupled Group Reward Optimization (DGRO), a general RL algorithm for LLM reasoning. On the one hand, DGRO decouples the traditional regularization coefficient into two independent hyperparameters: one scales the policy gradient term, and the other regulates the distance from the sampling policy. This decoupling not only enables precise control over balancing exploration and exploitation, but also can be seamlessly extended to Online Policy Mirror Descent (OPMD) algorithms in Kimi k1.5 and Direct Reward Optimization. On the other hand, we observe that reward variance significantly affects both convergence speed and final model performance. We conduct both theoretical analysis and extensive empirical validation to assess DGRO, including a detailed ablation study that investigates its performance and optimization dynamics. Experimental results show that DGRO achieves state-of-the-art performance on the Logic dataset with an average accuracy of 96.9\%, and demonstrates strong generalization across mathematical benchmarks.
Paper Structure (33 sections, 6 theorems, 88 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 33 sections, 6 theorems, 88 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Works
  4. Preliminary
  5. Decoupled Group Reward Optimization algorithm
  6. Theoretical analysis for using $\bar{r}(x)$ to approximate $V(x,\beta)$
  7. Gradient Behavior under Varying Reward Variance and Coefficient Settings
  8. Experiment
  9. DGRO Implementations
  10. Experimental Results
  11. Logic.
  12. Math.
  13. Ablation Study
  14. Reward variance.
  15. Influence of decoupled $\beta_1$ and $\beta_2$.
  16. ...and 18 more sections

Key Result

Lemma 3.1

Denote $\bar{r}(x)=\mathbb{E}_{y\sim \pi_{{\text{sam}}}(\cdot|x)}[r(x,y)]$, $V(x, \beta)=\beta\log\sum_y \pi_{\text{src}}(y \mid x) \exp \left(\frac{1}{\beta} r(x, y)\right)$. When $\pi_{{\text{sam}}}=\pi_{{\text{src}}}$, the following equation holds if and only if $\beta\to\infty$ or $r(x,y)=c(x)$

Figures (4)

  • Figure 1: Comparison of (a) accuracy, (b) response length, and (c) entropy for different reward ranges on the K&K dataset using DGRO. In (a), solid lines denote mean accuracy over three seeds, with shaded areas indicating variance. In (b) and (c), solid lines represent smoothed means, and shaded areas show raw measurements.
  • Figure 2: Comparison of (a) accuracy, (b) response length, and (c) entropy during training for different $(\beta_1, \beta_2)$ values.
  • Figure 3: Comparison of (a) accuracy, (b) response length, and (c) entropy for different models: DGRO with reward $\in [-5, 5]$ (red), DGRO with reward $\in [-3, 3]$ (yellow), Online DPO (green) and GRPO (blue). DGRO curves show mean performance over three seeds.
  • Figure 4: Entropy during training for math reasoning. The base model is DeepSeek-R1-Distill-Qwen-7B, and the training dataset is orz and deepscaler respectively.

Theorems & Definitions (6)

  • Lemma 3.1
  • Lemma 3.2
  • Theorem 3.3
  • Theorem 3.4
  • Theorem E.1
  • Lemma E.2