Diagnosing and Mitigating System Bias in Self-Rewarding RL

TL;DR

This work identifies a system bias in reinforcement learning with intrinsic rewards (RLIR), where the model overestimates high-confidence, correct rollouts, leading to biased and unstable reward estimation. It formalizes three metrics, $\rho_{noise}$, $\rho_{selfbias}$, and $\rho_{symbias}$, to diagnose this bias and shows how it impedes stable unlabeled data scaling. To mitigate bias, the authors propose reinforcement learning with ensembled rewards (RLER), which ensembles multiple policies to construct a unified reward space and employs adaptive soft-reward interpolation alongside confidence–disagreement balanced rollout selection, culminating in a model-merge step for deployment. Empirical results demonstrate that RLER reduces the bias metrics, delivers up to +13.6\% improvement over RLIR, and approaches RLVR performance, enabling stable scaling with unlabeled data and practical deployment via model merging. This approach offers a concrete, scalable path to leveraging large unlabeled corpora for high-capability reasoning systems while mitigating training instability.

Abstract

Reinforcement learning with verifiable rewards (RLVR) scales the reasoning ability of large language models (LLMs) but remains bottlenecked by limited labeled samples for continued data scaling. Reinforcement learning with intrinsic rewards (RLIR), where the policy model assigns rewards to its own rollouts, enables sustainable scaling in unlabeled settings, yet its performance and stability lag behind RLVR. We trace this gap to a system bias: the model tends to overestimate its high-confidence rollouts, leading to biased and unstable reward estimation. This bias accumulates as training progresses, with deviations from the oracle drifting toward over-reward, causing unstable training. We characterize this bias using three metrics: $ρ_{\text{noise}}$, $ρ_{\text{selfbias}}$, and $ρ_{\text{symbias}}$. We find that $ρ_{\text{noise}}$ and $ρ_{\text{symbias}}$ impact convergence, while $ρ_{\text{selfbias}}$ amplifies both correct and incorrect updates, leading to instability. To mitigate this, we propose reinforcement learning with ensembled rewards (RLER), which aggregates diverse models and adapts reward interpolation and rollout selection. Extensive experiments show that RLER improves by +13.6% over RLIR and is only 3.6% below RLVR, achieving stable scaling on unlabeled samples, making it highly applicable.

Figures (7)

• Figure 1: Flowchart of the process for reinforcement learning with intrinsic rewards (RLIR).
• Figure 2: Effects of $\rho_{\mathrm{noise}}$, $\rho_{\mathrm{symbias}}$, and $\rho_{\mathrm{selfbias}}$ during training on the arithmetic dataset.
• Figure 3: Training results of RLIR methods on arithmetic dataset.
• Figure 4: Training results of compared baselines and RLER on DAPO-Math-17K.
• Figure 5: Ablations of RLER (left) and reward interpolation (right). Left: Avg@8 on the test benchmarks for each ablation. Right: interpolation gain across interpolation variants.

Theorems & Definitions (1)

• Theorem 1