Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients

Omar El Mansouri, Mohamed El Amine Seddik, Salem Lahlou

TL;DR

This work tackles the vulnerability of RLHF/RLVR to noisy reward signals by introducing Noise-corrected GRPO and Dr.GRPO, which explicitly model reward corruption as a Bernoulli channel. By estimating flip rates and applying a Natarajan-inspired debiasing correction within a generalized objective, the approach yields unbiased gradient estimates and preserves monotonic policy improvement despite noisy rewards. The authors provide formal guarantees for both noiseless and noisy scenarios and validate the method on math and code tasks, showing tangible improvements (up to 6.7 percentage points on math and 1.5 on code). Overall, the framework bridges label-noise correction techniques from supervised learning with modern RLHF, offering a principled, practical path to robust deployment under realistic reward-noise conditions.

Abstract

Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitive to noise from inconsistent or erroneous rewards. Yet, the interaction between such noise and widely used group-based policy optimization methods remains underexplored. We introduce a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise. Our method applies noise correction after estimating reward flip probabilities to debias the learning signal, yielding provably unbiased gradient estimates. Theoretical analysis shows that group-based methods inherently mitigate individual-level noise, and our correction strategy amplifies this robustness. Empirically, we observe consistent improvements across math and code tasks when applying our noise correction to standard reward model usage, with particular gains of up to 6.7 percentage points in accuracy on math tasks and 1.5 on code tasks under realistic reward model conditions. This work bridges label-noise correction from supervised learning with modern RLHF, offering both theoretical insights and a practical algorithm for noisy real-world deployment.

Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients

TL;DR

This work tackles the vulnerability of RLHF/RLVR to noisy reward signals by introducing Noise-corrected GRPO and Dr.GRPO, which explicitly model reward corruption as a Bernoulli channel. By estimating flip rates and applying a Natarajan-inspired debiasing correction within a generalized objective, the approach yields unbiased gradient estimates and preserves monotonic policy improvement despite noisy rewards. The authors provide formal guarantees for both noiseless and noisy scenarios and validate the method on math and code tasks, showing tangible improvements (up to 6.7 percentage points on math and 1.5 on code). Overall, the framework bridges label-noise correction techniques from supervised learning with modern RLHF, offering a principled, practical path to robust deployment under realistic reward-noise conditions.

Abstract

Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitive to noise from inconsistent or erroneous rewards. Yet, the interaction between such noise and widely used group-based policy optimization methods remains underexplored. We introduce a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise. Our method applies noise correction after estimating reward flip probabilities to debias the learning signal, yielding provably unbiased gradient estimates. Theoretical analysis shows that group-based methods inherently mitigate individual-level noise, and our correction strategy amplifies this robustness. Empirically, we observe consistent improvements across math and code tasks when applying our noise correction to standard reward model usage, with particular gains of up to 6.7 percentage points in accuracy on math tasks and 1.5 on code tasks under realistic reward model conditions. This work bridges label-noise correction from supervised learning with modern RLHF, offering both theoretical insights and a practical algorithm for noisy real-world deployment.
Paper Structure (34 sections, 11 theorems, 81 equations, 1 figure, 5 tables, 1 algorithm)

This paper contains 34 sections, 11 theorems, 81 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Positioning within prior work.
  3. Our contributions are the following:
  4. Background
  5. Group-Relative Policy Optimisation:
  6. Done Right GRPO:
  7. Preliminaries on the noiseless case
  8. Noisy GRPO
  9. Binary corruption model.
  10. Stochastic representation:
  11. Expectation:
  12. Variance:
  13. Centered and normalized objective.
  14. Monotonic improvement in the noisy case:
  15. Difficulty bias noisy case:
  16. ...and 19 more sections

Key Result

Theorem 1

For any policy $\pi$,

Figures (1)

  • Figure 1: Effect of Label Noise on the Lower Bound. Noise flattens the penalization term.

Theorems & Definitions (12)

  • Definition 1: success probability
  • Theorem 1: Policy Improvement Guarantee
  • Theorem 2: Recursion for success probability
  • Proposition 1: Monotonic improvement
  • Theorem 3: Policy Improvement Guarantee
  • Theorem 4: Recursion in the noisy case
  • Proposition 2: Amplification in the noisy case
  • Proposition 3: Debiasing
  • Theorem 5
  • Proposition 4: Estimator of the true variance
  • ...and 2 more