Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Learning Dynamics of RLVR at the Edge of Competence

Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, Yuxin Chen

TL;DR

The paper investigates RLVR on long-horizon, multi-step compositional reasoning with transformers, revealing two distinct learning regimes governed by the difficulty spectrum: grokking plateaus when the horizon sequence includes abrupt jumps and a relay dynamics where gradient signals smoothly propagate from easy to hard tasks. It formalizes a minimal transformer setup with fixed one-step atomic skills and outcome-based rewards, develops a Fourier-analytic framework on finite groups to analyze trajectory-conditioned gradients, and provides convergence guarantees for short-horizon RL as well as mixed-difficulty curricula. The authors validate the theory with synthetic experiments showing the predicted grokking and relay behaviors and highlight data-mixture design as a scalable lever to extend performance toward the edge of competence. Together, the results offer mechanistic insight into RLVR’s effectiveness and suggest practical data-design strategies to achieve scalable gains in long-horizon reasoning.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theory of the training dynamics of RL for transformers on compositional reasoning tasks. Our theory characterizes how the effectiveness of RLVR is governed by the smoothness of the difficulty spectrum. When data contains abrupt discontinuities in difficulty, learning undergoes grokking-type phase transitions, producing prolonged plateaus before progress recurs. In contrast, a smooth difficulty spectrum leads to a relay effect: persistent gradient signals on easier problems elevate the model's capabilities to the point where harder ones become tractable, resulting in steady and continuous improvement. Our theory explains how RLVR can improve performance at the edge of competence, and suggests that appropriately designed data mixtures can yield scalable gains. As a technical contribution, our analysis develops and adapts tools from Fourier analysis on finite groups to our setting. We validate the predicted mechanisms empirically via synthetic experiments.

On the Learning Dynamics of RLVR at the Edge of Competence

TL;DR

The paper investigates RLVR on long-horizon, multi-step compositional reasoning with transformers, revealing two distinct learning regimes governed by the difficulty spectrum: grokking plateaus when the horizon sequence includes abrupt jumps and a relay dynamics where gradient signals smoothly propagate from easy to hard tasks. It formalizes a minimal transformer setup with fixed one-step atomic skills and outcome-based rewards, develops a Fourier-analytic framework on finite groups to analyze trajectory-conditioned gradients, and provides convergence guarantees for short-horizon RL as well as mixed-difficulty curricula. The authors validate the theory with synthetic experiments showing the predicted grokking and relay behaviors and highlight data-mixture design as a scalable lever to extend performance toward the edge of competence. Together, the results offer mechanistic insight into RLVR’s effectiveness and suggest practical data-design strategies to achieve scalable gains in long-horizon reasoning.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theory of the training dynamics of RL for transformers on compositional reasoning tasks. Our theory characterizes how the effectiveness of RLVR is governed by the smoothness of the difficulty spectrum. When data contains abrupt discontinuities in difficulty, learning undergoes grokking-type phase transitions, producing prolonged plateaus before progress recurs. In contrast, a smooth difficulty spectrum leads to a relay effect: persistent gradient signals on easier problems elevate the model's capabilities to the point where harder ones become tractable, resulting in steady and continuous improvement. Our theory explains how RLVR can improve performance at the edge of competence, and suggests that appropriately designed data mixtures can yield scalable gains. As a technical contribution, our analysis develops and adapts tools from Fourier analysis on finite groups to our setting. We validate the predicted mechanisms empirically via synthetic experiments.
Paper Structure (91 sections, 64 theorems, 187 equations, 3 figures)

This paper contains 91 sections, 64 theorems, 187 equations, 3 figures.

Table of Contents

  1. Introduction
  2. An optimization perspective for RL at the edge of competence.
  3. Related Work
  4. Empirical understanding of RLVR mechanisms.
  5. Theory of RL training for LLM.
  6. Grokking in supervised learning and RL.
  7. Problem Setup
  8. Notation.
  9. Compositional Reasoning
  10. Embeddings and tokenizer.
  11. Transformer Architecture
  12. Attention Layer.
  13. MLP layer.
  14. Induced next-state distribution.
  15. Pretrained Atomic Skills
  16. ...and 76 more sections

Key Result

Proposition 3.1

Under Assumption assum-group-assump-x, if the MLP weight $W$ satisfies eq:choice-mlp-compact-pos--eq:choice-mlp-compact-zero, given any $Q$, for any $y_0\in\mathcal{Y}$ and $G^{1}=(g_1)$ with $g_1\in\mathcal{G}$, we have

Figures (3)

  • Figure 1: Reward-growth dynamics in mixed-difficulty RL. A schematic illustration of the reward growth rate $\mathrm{d}r/\mathrm{d}t$ and $r(t)$ for mixed-difficulty RL, demonstrating how the difficulty ratio $R=L_{k+1}/L_k$ changes the learning dynamics at the edge of model's competence, which yields either grokking-type phase transitions or smooth relays.
  • Figure 2: Average reward and peak attention hit rate during fixed-length RL training.
  • Figure 3: Average reward during mixed-length RL training under different difficulty ratios.

Theorems & Definitions (106)

  • Definition 3.1: $L$-step compositional reasoning
  • Remark 3.1
  • Definition 3.2: Reasoning problems
  • Definition 3.3: Data distribution $\mathcal{D}^{L}$
  • Definition 3.4: One-layer transformer Vaswani2017AttentionIA
  • Remark 3.2
  • Proposition 3.1
  • Remark 3.3
  • Theorem 4.1: RL for short-horizon problems
  • Proposition 4.1: Exponentially flat region
  • ...and 96 more