Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution

Shengda Fan, Xuyan Ye, Yankai Lin

TL;DR

The paper tackles instability in self-evolving LLMs caused by non-stationary, solver-dependent rewards and noisy self-labels. It introduces DARC, a two-stage framework that decouples question generation from solver training and employs asymmetric self-distillation with a document-augmented teacher to supervise a question-only student, all grounded in external corpora. Across multiple backbones and reasoning benchmarks, DARC delivers consistent gains (average +10.9 points) and approaches the performance of fully supervised systems without human annotations, while demonstrating stability in training dynamics and strong generalization. Theoretical analysis explains why coupling induces instability and why decoupling stabilizes optimization, with empirical evidence of solver-question difficulty alignment being largely solver-agnostic and transferable across models. This work offers a practical, model-agnostic path toward robust self-evolution of LLMs that reduces reliance on human data.

Abstract

Self-play with large language models has emerged as a promising paradigm for achieving self-improving artificial intelligence. However, existing self-play frameworks often suffer from optimization instability, due to (i) non-stationary objectives induced by solver-dependent reward feedback for the Questioner, and (ii) bootstrapping errors from self-generated pseudo-labels used to supervise the Solver. To mitigate these challenges, we introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process. First, we train the Questioner to synthesize difficulty-calibrated questions, conditioned on explicit difficulty levels and external corpora. Second, we train the Solver with an asymmetric self-distillation mechanism, where a document-augmented teacher generates high-quality pseudo-labels to supervise the student Solver that lacks document access. Empirical results demonstrate that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models. Moreover, DARC consistently outperforms all baselines and approaches the performance of fully supervised models without relying on human annotations.The code is available at https://github.com/RUCBM/DARC.

DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution

TL;DR

The paper tackles instability in self-evolving LLMs caused by non-stationary, solver-dependent rewards and noisy self-labels. It introduces DARC, a two-stage framework that decouples question generation from solver training and employs asymmetric self-distillation with a document-augmented teacher to supervise a question-only student, all grounded in external corpora. Across multiple backbones and reasoning benchmarks, DARC delivers consistent gains (average +10.9 points) and approaches the performance of fully supervised systems without human annotations, while demonstrating stability in training dynamics and strong generalization. Theoretical analysis explains why coupling induces instability and why decoupling stabilizes optimization, with empirical evidence of solver-question difficulty alignment being largely solver-agnostic and transferable across models. This work offers a practical, model-agnostic path toward robust self-evolution of LLMs that reduces reliance on human data.

Abstract

Self-play with large language models has emerged as a promising paradigm for achieving self-improving artificial intelligence. However, existing self-play frameworks often suffer from optimization instability, due to (i) non-stationary objectives induced by solver-dependent reward feedback for the Questioner, and (ii) bootstrapping errors from self-generated pseudo-labels used to supervise the Solver. To mitigate these challenges, we introduce DARC (Decoupled Asymmetric Reasoning Curriculum), a two-stage framework that stabilizes the self-evolution process. First, we train the Questioner to synthesize difficulty-calibrated questions, conditioned on explicit difficulty levels and external corpora. Second, we train the Solver with an asymmetric self-distillation mechanism, where a document-augmented teacher generates high-quality pseudo-labels to supervise the student Solver that lacks document access. Empirical results demonstrate that DARC is model-agnostic, yielding an average improvement of 10.9 points across nine reasoning benchmarks and three backbone models. Moreover, DARC consistently outperforms all baselines and approaches the performance of fully supervised models without relying on human annotations.The code is available at https://github.com/RUCBM/DARC.
Paper Structure (38 sections, 1 theorem, 16 equations, 9 figures, 5 tables)

This paper contains 38 sections, 1 theorem, 16 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for LLM Reasoning.
  4. Self-Play for LLM Self-Evolution.
  5. Data Synthesis for LLM Training.
  6. Methodology
  7. Questioner: Difficulty-Aware Generation
  8. Solver: Offline Curriculum Learning via Asymmetric Self-Distillation
  9. Theoretical Analysis
  10. Toy Model of Coupled Self-Play
  11. Setup.
  12. Structural Instability in Coupled Self-Play
  13. Proof sketch.
  14. Interpretation.
  15. Empirical Analysis.
  16. ...and 23 more sections

Key Result

Theorem 1

Consider the toy model above. Assume the Solver updates according to Let the Questioner perform a single gradient ascent step where $\partial J_t(\theta_t)$ denotes a (sub)gradient and $\alpha > 0$ is sufficiently small. Then, for any $\delta$ satisfying if $\pi_{\theta_t}$ concentrates around $\tau=\phi_t+\delta$ (e.g., $\Pr_{\tau\sim\pi_{\theta_t}}[|\tau-(\phi_t+\delta)|\le \epsilon]\approx 1

Figures (9)

  • Figure 1: Comparison between DARC and previous coupled self-play methods.
  • Figure 2: Illustration of the two-stage DARC framework. In the first stage (the upper half), the Questioner learns to generate questions matching specified difficulty $\tau$ with a difficulty-anchored reward. In the second stage (the lower half), the Solver is trained on an offline curriculum with an answer correctness reward.
  • Figure 3: Cross-Iteration Accuracy Heatmap in Coupled Self-Play. Golden label is annotated by DeepSeek-3.2 liu2025deepseek and only for analytical purposes.
  • Figure 4: Training dynamics of DARC under curriculum-ordered training and random shuffling on the same offline question set. The validation reward in (b) is evaluated on the Math12K test set.
  • Figure 5: Accuracy of different Solvers on questions generated under different difficulty levels.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 1: Gradient direction reversal under coupling