Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe

TL;DR

This paper introduces SOAR, a self-improvement framework that enables an LLM to teach itself by generating a stepping-stone curriculum through asymmetric teacher–student meta-RL. By grounding the teacher's rewards in measured student progress on hard problems, SOAR sidesteps the need for intrinsic rewards and circumvents learning plateaus caused by sparse signals. The results show that grounded, bilevel meta-RL yields stable, diverse teacher policies and generates effective synthetic questions that unlock learning on extremely difficult datasets, with notable transfer to out-of-domain tasks. This work demonstrates that a model's latent pedagogical capabilities can be elicited without human-curated data or direct solution to the hard problems, offering a principled route to expanding the learnability frontier in reasoning tasks.

Abstract

Can a model learn to escape its own learning plateau? Reinforcement learning methods for finetuning large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? To explore this, we design SOAR: A self-improvement framework designed to surface these pedagogical signals through meta-RL. A teacher copy of the model proposes synthetic problems for a student copy, and is rewarded with its improvement on a small subset of hard problems. Critically, SOAR grounds the curriculum in measured student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of mathematical benchmarks (0/128 success) reveals three core findings. First, we show that it is possible to realize bi-level meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful stepping stones. Second, grounded rewards outperform intrinsic reward schemes used in prior LLM self-play, reliably avoiding the instability and diversity collapse modes they typically exhibit. Third, analyzing the generated questions reveals that structural quality and well-posedness are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to actually solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data.

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

TL;DR

This paper introduces SOAR, a self-improvement framework that enables an LLM to teach itself by generating a stepping-stone curriculum through asymmetric teacher–student meta-RL. By grounding the teacher's rewards in measured student progress on hard problems, SOAR sidesteps the need for intrinsic rewards and circumvents learning plateaus caused by sparse signals. The results show that grounded, bilevel meta-RL yields stable, diverse teacher policies and generates effective synthetic questions that unlock learning on extremely difficult datasets, with notable transfer to out-of-domain tasks. This work demonstrates that a model's latent pedagogical capabilities can be elicited without human-curated data or direct solution to the hard problems, offering a principled route to expanding the learnability frontier in reasoning tasks.

Abstract

Can a model learn to escape its own learning plateau? Reinforcement learning methods for finetuning large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? To explore this, we design SOAR: A self-improvement framework designed to surface these pedagogical signals through meta-RL. A teacher copy of the model proposes synthetic problems for a student copy, and is rewarded with its improvement on a small subset of hard problems. Critically, SOAR grounds the curriculum in measured student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of mathematical benchmarks (0/128 success) reveals three core findings. First, we show that it is possible to realize bi-level meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful stepping stones. Second, grounded rewards outperform intrinsic reward schemes used in prior LLM self-play, reliably avoiding the instability and diversity collapse modes they typically exhibit. Third, analyzing the generated questions reveals that structural quality and well-posedness are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to actually solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data.
Paper Structure (46 sections, 1 theorem, 7 equations, 18 figures, 7 tables, 1 algorithm)

This paper contains 46 sections, 1 theorem, 7 equations, 18 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Curriculum Learning in RL:
  4. Self-Play and Teacher-Student Setups:
  5. Intrinsic Rewards versus Bilevel Optimization
  6. Method
  7. Overview
  8. Outer Loop: Teacher Training
  9. Inner Loop: Student Training
  10. Experiment Setup
  11. Models and Datasets
  12. Teacher-student training
  13. Evaluation
  14. Baselines
  15. Metrics
  16. ...and 31 more sections

Key Result

Proposition 1

Let $\pi_0(z)$ be a proposal distribution over some random variable $z$. Let $S$ be a set of "accepted" values of $z$, and assume $\pi_0(S)>0$. Let be the distribution on $z$ obtained by rejection sampling, namely, sampling $z$ from $\pi_0$ until $z\in S$. Let $R(z)$ be some reward function on $z$. Then the RLOO update on $\pi$ can be computed from gradient of $\pi_0$ only. Namely, for any $g$-tu

Figures (18)

  • Figure 1: Learning on hard problems by self-generating a curriculum. We introduce SOAR: A meta-RL framework for improving on difficult datasets where performance plateaus. (left) We initialize asymmetric teacher and student models from the same base model. The teacher generates synthetic problems for the student to train on with RL, and is rewarded by the student's measurable improvement on a small subset of the real, ground-truth problems. (right) RL training on problems generated with SOAR, using grounded teacher rewards, outperforms direct training on the hard problems and enables the student to break out of the performance plateau.
  • Figure 2: The SOAR meta-RL Loop. The teacher and student are initialized from the same model. In the outer RL loop the teacher generates candidate question-answer pairs that are partitioned into datasets. In the inner RL loop, the student is trained for 10 steps on the candidate problems and evaluated on sampled hard problems. The teacher is rewarded based on the resulting student improvement over the student baseline, grounding the synthetic curriculum in real learning progress.
  • Figure 3: Performance on MATH and HARP fail@128 (improvement over Hard-Only). Synthetic problems generated with SOAR (PQ) and inference with the promoted student (PS) outperform direct training on fail@128 train sets (Hard-Only), and sampling from teachers trained with intrinsic rewards (Intrinsic-T). Performance is reported as the delta over Hard-Only. For reference, Hard-Only MATH pass@$k$ for $k \in \{1,4,8,16,32\}$ is $\{0.5, 1.7, 3.2, 5.7, 9.6\}$. Hard-Only training curves are shown in Figure \ref{['fig:teacher-ablation']}; absolute performance for all methods, and further evaluations, are in Tables \ref{['tab:app-promotion-results-math']}-\ref{['tab:app-promotion-results-harp']}. Shaded regions are $\pm$ 1 SD over 6-12 seeds nested across teacher/student training (see \ref{['app:seeds']}).
  • Figure 4: Transfer performance to OlympiadBench fail@128 subset (improvement over Hard-Only). Questions optimized for MATH and HARP transfer to a held-out dataset. Performance is reported as the delta over Hard-Only; absolute performance, including PS evaluation, is in Table \ref{['tab:app-promotion-results-olympiad']}.
  • Figure 5: Grounded rewards lead to more stable teacher policies. We evaluate trained teacher policies by sampling questions and training fresh students. (Left) Test pass@32 comparison between students trained with questions sampled from Grounded-T and Base-T (Hard-Only also shown for reference). Grounded-T outperforms Base-T and exhibits more stable student trajectories. (Right) Pass@32 trajectories for fresh students trained with individual Grounded-T teacher seeds (red) and Intrinsic-T teacher seeds (green). Questions from Grounded-T yield consistent student trajectories, whereas Intrinsic-T exhibits higher variance across teachers, including a failure mode where I-T (1) causes student collapse. Shading shows $\pm 1$ SD. Curves for other pass@k and OlympiadBench are in Figures \ref{['fig:teacher-ablation-MATH']}-\ref{['fig:teacher-ablation-olympiad']}.
  • ...and 13 more figures

Theorems & Definitions (2)

  • Proposition 1: RLOO update with rejection sampling
  • proof