Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

Tiehua Mei, Minxuan Lv, Leiyu Pan, Zhenpeng Su, Hongru Hou, Hengrui Chen, Ao Xu, Deqing Yang

TL;DR

It is observed that better reasoning are better teachers: high-quality solutions serve as more effective demonstrations than low-quality ones, and it is shown that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed Evidence Gain.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that get correct answers by chance. We observe that better reasoning are better teachers: high-quality solutions serve as more effective demonstrations than low-quality ones. We term this teaching ability Demonstration Utility, and show that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed Evidence Gain. To employ this signal during training, we introduce In-Context RLVR. By Bayesian analysis, we show that this objective implicitly reweights rewards by Evidence Gain, assigning higher weights to high-quality traces and lower weights to low-quality ones, without requiring costly computation or external evaluators. Experiments on mathematical benchmarks show improvements in both accuracy and reasoning quality over standard RLVR.

Good Reasoning Makes Good Demonstrations: Implicit Reasoning Quality Supervision via In-Context Reinforcement Learning

TL;DR

It is observed that better reasoning are better teachers: high-quality solutions serve as more effective demonstrations than low-quality ones, and it is shown that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed Evidence Gain.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in large language models but treats all correct solutions equally, potentially reinforcing flawed traces that get correct answers by chance. We observe that better reasoning are better teachers: high-quality solutions serve as more effective demonstrations than low-quality ones. We term this teaching ability Demonstration Utility, and show that the policy model's own in-context learning ability provides an efficient way to measure it, yielding a quality signal termed Evidence Gain. To employ this signal during training, we introduce In-Context RLVR. By Bayesian analysis, we show that this objective implicitly reweights rewards by Evidence Gain, assigning higher weights to high-quality traces and lower weights to low-quality ones, without requiring costly computation or external evaluators. Experiments on mathematical benchmarks show improvements in both accuracy and reasoning quality over standard RLVR.
Paper Structure (48 sections, 3 theorems, 16 equations, 7 figures, 3 tables)

This paper contains 48 sections, 3 theorems, 16 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Demonstration Utility as Global Quality Signal.
  3. Implicit Reward Reweighting via In-Context RLVR.
  4. Contributions.
  5. Evidence Gain as Quality Measure
  6. Definition.
  7. Empirical Validation.
  8. Implicit Reward Reweighting
  9. Experiments
  10. Setup
  11. Dataset.
  12. Baselines.
  13. Training and Evaluation.
  14. Main Results
  15. Analysis
  16. ...and 33 more sections

Key Result

Lemma E.2

Under Assumption assump:independence, the conditioned policy admits the decomposition:

Figures (7)

  • Figure 1: Mean Evidence Gain by quality score with 95% confidence intervals on two models.
  • Figure 2: Training dynamics of Evidence Gain, quality score, and their correlation ($\rho$) across training steps.
  • Figure 3: Spearman correlation between Evidence Gain and individual quality dimensions, alongside mean scores for each dimension. Evaluated on DeepSeek-R1-Distill-Qwen-1.5B with the same setup as Section \ref{['sec:evidence_gain']}.
  • Figure 4: Spearman correlation matrix between DeepSeek-V3.2 quality scores and four human expert ratings on 100 sampled reasoning traces, together with inter rater correlations among experts.
  • Figure 5: Training dynamics across 1.5B and 7B models. IC-DAPO variants consistently outperform DAPO on AIME24 and AIME25 while maintaining stable entropy throughout training.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Lemma E.2: Bayesian Identity
  • proof
  • Theorem E.3: Implicit Reweighting
  • proof
  • Theorem E.4: Reweighting Equivalence