Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

Teng Pan, Yuchen Yan, Zixuan Wang, Ruiqing Zhang, Gaiyang Han, Wanqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen

Abstract

Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9\% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55\% to over 85\%, confirming that both capabilities genuinely co-evolve.

CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

Abstract

Label-free reinforcement learning enables large language models to improve reasoning capabilities without ground-truth supervision, typically by treating majority-voted answers as pseudo-labels. However, we identify a critical failure mode: as training maximizes self-consistency, output diversity collapses, causing the model to confidently reinforce systematic errors that evade detection. We term this the consensus trap. To escape it, we propose CoVerRL, a framework where a single model alternates between generator and verifier roles, with each capability bootstrapping the other. Majority voting provides noisy but informative supervision for training the verifier, while the improving verifier progressively filters self-consistent errors from pseudo-labels. This co-evolution creates a virtuous cycle that maintains high reward accuracy throughout training. Experiments across Qwen and Llama model families demonstrate that CoVerRL outperforms label-free baselines by 4.7-5.9\% on mathematical reasoning benchmarks. Moreover, self-verification accuracy improves from around 55\% to over 85\%, confirming that both capabilities genuinely co-evolve.
Paper Structure (49 sections, 95 equations, 10 figures, 10 tables)

This paper contains 49 sections, 95 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Self-Evolution
  4. LLM Self-Verification and Self-Correction
  5. Method
  6. Pseudo-Label Generation
  7. Majority Voting.
  8. Self-consistent Verification Filtering.
  9. Online Dual-Role Co-Evolution
  10. Contrastive Verifier Training.
  11. Self-Correction.
  12. Reward Design.
  13. Answer-Anchored GRPO.
  14. The Co-Evolution Cycle.
  15. Experiments
  16. ...and 34 more sections

Figures (10)

  • Figure 1: Training dynamics of TTRL and CoVerRL on Qwen3-1.7B-Base. CoVerRL maintains high reward accuracy while steadily improving label accuracy, whereas TTRL suffers from reward accuracy degradation as output diversity collapses.
  • Figure 2: Overview of the CoVerRL framework. The pipeline illustrates the Pseudo-Label Generation process and the Online Dual-Role Co-Evolution strategy, facilitating a mutual bootstrapping process where generation and verification capabilities jointly evolve.
  • Figure 3: Co-evolution dynamics across three model backbones. As training progresses, verification accuracy improves through contrastive learning, which in turn enables better filtering of self-consistent errors and leads to higher label accuracy than TTRL.
  • Figure 4: Evolution of the verification capability of the Qwen3-1.7B-Base. We report the True Negative Rate (TNR) and the False Positive Rate (FPR).
  • Figure 5: Reward accuracy across generation, verification, and self-correction roles during training on Llama-3.2-3B-Instruct.
  • ...and 5 more figures