Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Reward Miscalibration of GRPO in Agentic RL

Jingyu Liu, Xiaopeng Wu, Jingquan Peng, Kehan Chen, Chuan Yu, Lizhong Ding, Yong Liu

TL;DR

The paper addresses reward miscalibration in long-horizon agentic RL and argues that gradient coupling from highly similar training samples, not reward miscalibration alone, drives persistent flawed actions. It derives that the expected advantage for a flawed action is negative, $A_i = q r (q-1)$, and demonstrates that the squeezing effect does not eliminate this issue. To combat cross-sample interference, they introduce Generative Classification Disentanglement (GCD), training the actor to classify actions as good or bad to disentangle their embeddings, complemented by prompt-based corrections to move dangerous actions into a safe region. Empirically, GCD improves out-domain generalization on ALFWorld and ScienceWorld and is shown to robustly reduce gradient coupling, suggesting a practical path to stabilize outcome-based RL in complex, multi-turn tasks.

Abstract

Building autonomous agents capable of solving long-horizon, real-world tasks has garnered significant research interest. But outcome based rewards may cause reward miscalibration which means it might mistakenly allocate positive reward to flawed middle steps which is regarded as the key reason making the bad actions being reinforced during training. However we reveal that outcome based reward ensures expected negative advantage for those flawed middle steps, which means the flawed actions should be punished during training. Even accounting for the ``squeezing effect", the probability mass of good actions should increase and the actor should gradually get rid of harmful actions. This shows that flawed actions should be punished during training. We further identify gradient coupling between similar samples as a key issue in agentic RL, the input prompt is extremely similar and the output action space is limited, therefore during training, gradients from well-performing samples can inadvertently strengthen suboptimal or incorrect actions due to similar input observation and output actions. We show that with gradient coupling, some flawed actions might be enhanced. To address this, we propose training the actor to classify good or bad actions to separate the embedding of good/bad actions and alleviate the gradient interference, extensive experiments shows its effectiveness.

Rethinking Reward Miscalibration of GRPO in Agentic RL

TL;DR

The paper addresses reward miscalibration in long-horizon agentic RL and argues that gradient coupling from highly similar training samples, not reward miscalibration alone, drives persistent flawed actions. It derives that the expected advantage for a flawed action is negative, , and demonstrates that the squeezing effect does not eliminate this issue. To combat cross-sample interference, they introduce Generative Classification Disentanglement (GCD), training the actor to classify actions as good or bad to disentangle their embeddings, complemented by prompt-based corrections to move dangerous actions into a safe region. Empirically, GCD improves out-domain generalization on ALFWorld and ScienceWorld and is shown to robustly reduce gradient coupling, suggesting a practical path to stabilize outcome-based RL in complex, multi-turn tasks.

Abstract

Building autonomous agents capable of solving long-horizon, real-world tasks has garnered significant research interest. But outcome based rewards may cause reward miscalibration which means it might mistakenly allocate positive reward to flawed middle steps which is regarded as the key reason making the bad actions being reinforced during training. However we reveal that outcome based reward ensures expected negative advantage for those flawed middle steps, which means the flawed actions should be punished during training. Even accounting for the ``squeezing effect", the probability mass of good actions should increase and the actor should gradually get rid of harmful actions. This shows that flawed actions should be punished during training. We further identify gradient coupling between similar samples as a key issue in agentic RL, the input prompt is extremely similar and the output action space is limited, therefore during training, gradients from well-performing samples can inadvertently strengthen suboptimal or incorrect actions due to similar input observation and output actions. We show that with gradient coupling, some flawed actions might be enhanced. To address this, we propose training the actor to classify good or bad actions to separate the embedding of good/bad actions and alleviate the gradient interference, extensive experiments shows its effectiveness.

Paper Structure

This paper contains 21 sections, 3 theorems, 37 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GRPO in Agentic Task
  4. The Expected Advantage
  5. The Influence of Similar Sample Gradients
  6. The Learning Dynamic
  7. Methodology
  8. Weaken the Gradient Coupling
  9. Escaping the "Danger Zone" with Prompt-based Correction
  10. Experiments
  11. Main Results
  12. Ablation Study
  13. The gradient coupling between samples
  14. Conclusion
  15. Proofs
  16. ...and 6 more sections

Key Result

Lemma 3.1

For a policy $\pi_{\theta}$ with probability $q$ of conducting action $a_i$, conducting action $a_i$ brings risk $r$, then the expected advantage for action $a_i$ is where $\mathbb{E}_{\pi_{\theta}} A_{i}$ stands for the expected advantage of action $a_i$ (we remove the std for convenience).

Figures (11)

  • Figure 1: The consistency of the repeat actions, we can observe that the model gradually being more confident in some repetition (increasing high consistency percentage). High consistency means 5 repetition in 10 trials
  • Figure 2: Repetition in success and failure trajectories
  • Figure 3: The gradient coupling
  • Figure 4: The change of advantage
  • Figure 5: The dynamic of consistency in GRPO, we can observe that the consistency of some high probability bad actions increases with training. Action labels are determined by the consistency of 5 judgments from Deepseek V3.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Lemma 3.1
  • Theorem 3.2
  • Theorem A.1