Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models

TL;DR

The paper addresses the scalability bottleneck of RL with verifiable rewards by introducing Co-rewarding, a self-supervised RL framework that prevents training collapse through cross-view supervision. It presents two instantiations: Co-rewarding-I uses cross-view contrastive agreement on semantically equivalent questions, and Co-rewarding-II uses a slowly updated teacher to provide insulated pseudo-labels. Empirically, Co-rewarding improves stability and performance across multiple math, coding, and multi-task benchmarks, often surpassing self-reward baselines and, in some cases, even GT-reward signals, with standout results such as GSM8K Pass@1 of 94.01% on Qwen3-8B-Base. The work demonstrates that invariance-based, cross-view supervision can unlock robust reasoning in large language models without labeled data, offering a scalable path for self-supervised RL in real-world tasks.

Abstract

While reinforcement learning with verifiable rewards (RLVR) is effective to improve the reasoning ability of large language models (LLMs), its reliance on human-annotated labels leads to the scaling up dilemma, especially for complex tasks. Recent self-rewarding methods investigate a label-free alternative to unlock the reasoning capabilities of LLMs, yet they frequently encounter the non-negligible training collapse issue, as the single-view supervision signal easily forms the self-consistent illusion, yielding the reward hacking. Inspired by the success of self-supervised learning, we propose \textit{Co-rewarding}, a novel self-supervised RL framework that improves training stability by seeking complementary supervision from another views. Specifically, we instantiate Co-rewarding in two ways: (1) \textit{Co-rewarding-I} is a data-side instantiation that derives reward signals from contrastive agreement across semantically analogous questions; and (2) \textit{Co-rewarding-II} is a model-side instantiation that maintains a slowly-updated reference teacher with pseudo labels to realize self-distillation. Intuitively, such instantiations introduce different levels of discrepancy to increase the difficulty of training collapse on trivial reasoning solutions. Empirically, Co-rewarding exhibits stable training across various setups, and outperforms other self-rewarding baselines by $+3.31\%$ improvements on average on multiple mathematical reasoning benchmarks, especially by $+7.49\%$ on Llama-3.2-3B-Instruct. Notably, Co-rewarding reaches or even surpasses RLVR with ground-truth (GT) label in several cases, such as a Pass@$1$ of $94.01\%$ on GSM8K with Qwen3-8B-Base remarkably higher than GT. Our code is publicly available at https://github.com/tmlr-group/Co-rewarding.

Figures (12)

• Figure 1: Performance overview. Reasoning comparison of Pass@1 value and validation curves. Our Co-rewarding achieves better and more stable (without collapse) training than other baselines.
• Figure 2: Illustration of Co-rewarding framework: Unlike single-view methods that rely only on internal reward signal on original question (a), Co-rewarding introduces complementary supervision. On the data side (b), paraphrased questions yield pseudo-labels for cross-reference. On the model side (c), teacher model isolated from current policy provides stabilized pseudo-labels for updates.
• Figure 3: Performance curves comparison on validation set. Top: Qwen3-1.7B-Base and Qwen2.5-7B trained on the MATH set. Bottom: Qwen3-8B-Base and Llama-3.2-3B-Instruct trained on the DAPO-14k set.
• Figure 4: Detailed performance of MMLU-Pro with Qwen3-8B-Base trained on DAPO-14k. More results refer to Appendix \ref{['appe:Detail_MMLU-Pro']}.
• Figure 5: Performance and Stability on GSM8K and AMC. The gains of Co-rewarding arise from its training stability, which supports continuous improvements throughout learning.