R-Zero: Self-Evolving Reasoning LLM from Zero Data

TL;DR

<3-5 sentence high-level summary> R-Zero tackles the bottleneck of requiring large volumes of human-curated data to train self-evolving LLMs by introducing a fully autonomous co-evolution framework. It deploys two roles, a Challenger and a Solver, trained with Group Relative Policy Optimization and reinforced by verifiable rewards to generate and learn from their own tasks, starting from zero external data. Across multiple backbones, iterative self-improvement yields notable gains in math and general-domain reasoning and demonstrates synergy with supervised fine-tuning, while also revealing stability limits and data-quality challenges inherent to fully self-generated curricula. This approach points to a scalable path toward autonomous, self-improving reasoning systems without reliance on human-labeledTask datasets.

Abstract

Self-evolving Large Language Models (LLMs) offer a scalable path toward super-intelligence by autonomously generating, refining, and learning from their own experiences. However, existing methods for training such models still rely heavily on vast human-curated tasks and labels, typically via fine-tuning or reinforcement learning, which poses a fundamental bottleneck to advancing AI systems toward capabilities beyond human intelligence. To overcome this limitation, we introduce R-Zero, a fully autonomous framework that generates its own training data from scratch. Starting from a single base LLM, R-Zero initializes two independent models with distinct roles, a Challenger and a Solver. These models are optimized separately and co-evolve through interaction: the Challenger is rewarded for proposing tasks near the edge of the Solver capability, and the Solver is rewarded for solving increasingly challenging tasks posed by the Challenger. This process yields a targeted, self-improving curriculum without any pre-existing tasks and labels. Empirically, R-Zero substantially improves reasoning capability across different backbone LLMs, e.g., boosting the Qwen3-4B-Base by +6.49 on math-reasoning benchmarks and +7.54 on general-domain reasoning benchmarks.

Figures (4)

• Figure 1: (Left): R-Zero employs a co-evolutionary loop between Challenger and Solver. (Right): R-Zero achieves strong benchmark gains without any pre-existing tasks or human labels.
• Figure 2: An overview of our R-Zero framework, which illustrates the co-evolution of the Challenger and the Solver. Top: In the Challenger training phase, the Challenger is trained via GRPO to generate difficult questions. The reward signal is derived from the uncertainty for the frozen Solver, which is measured by the self-consistency of its multiple generated answers. Bottom: In the Solver training phase, the Solver is fine-tuned with GRPO on a filtered set of these challenging questions generated by the now-frozen Challenger, using the pseudo-labels voted by itself.
• Figure 3: Performance of R-Zero when combined with supervised fine-tuning. The dashed line represents the baseline of fine-tuning the base model on labelled data alone, showing that our iterative method provides a better initialization.
• Figure 4: Math performance across different iteration times and model scales. The star markers indicate the peak performance for each model size.