Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Yu Li, Tian Lan, Zhengling Qi

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at \href{https://github.com/Skylanding/BiCC}{https://github.com/Skylanding/BiCC}.

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Abstract

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at \href{https://github.com/Skylanding/BiCC}{https://github.com/Skylanding/BiCC}.
Paper Structure (43 sections, 7 theorems, 44 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 43 sections, 7 theorems, 44 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Reinforcement Learning for LLM reasoning
  4. Policy Gradient with Variance Reduction
  5. Another Look at GRPO and Variance Reduction
  6. Contrastive Reformulation of GRPO Objective
  7. Approximating Optimal Baseline under Importance Sampling
  8. Method
  9. Bilateral Context Conditioning
  10. Reward-Confidence Correction
  11. Result
  12. Experimental Setup
  13. Main Results
  14. Ablation and Analysis
  15. Related Work
  16. ...and 28 more sections

Key Result

Proposition 1.1

Under binary rewards $r_i \in \{0, 1\}$, the group-normalized advantages reduce to: where $\hat{p} = G^+/G$ is the proportion of correct samples.

Figures (4)

  • Figure 1: Overview of the proposed methods. The standard GRPO pipeline processes query $q$ through the policy model to generate $G$ outputs, which are scored by reward model and normalized to compute advantages for policy gradient update. BiCC partitions outputs by reward and conditions each sample on opposite-partition outputs, enabling cross-partition information flow where right and wrong attempts inform each other. RCC computes covariance between rewards and log-probability shifts to correct the advantage estimation.
  • Figure 2: Reward-confidence correlation analysis. $\text{Cov}(R, \delta)$ increases throughout training for both models. Distribution of $\delta$ at training steps 2.5k--3k shows clear separation by reward, with mean separation $\Delta_\mu = 0.27$ for Qwen3-4B and $0.56$ for Phi-4-mini.
  • Figure 3: Training and evaluation performance.
  • Figure 4: Evolution of $\delta$ distributions during training. The separation between $R=0$ and $R=1$ partitions widens as training progresses, reflecting the model's increasing ability to assign higher probability to correct outputs.

Theorems & Definitions (14)

  • proof
  • Proposition 1.1
  • proof
  • Corollary 1.2
  • Lemma 1.3
  • proof
  • Theorem 1.4
  • proof
  • Corollary 1.5
  • proof
  • ...and 4 more