RL for Reasoning by Adaptively Revealing Rationales

TL;DR

This work investigates this intermediate regime between SFT and RL and argues that per-sample curriculum learning is more than a trade-off between efficiency and generality--it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize.

Abstract

Learning in the combinatorially large output space of sequence generation problems is challenging as providing expert demonstrations scales poorly with sequence length, and RL struggles with sparse rewards. Between dense demonstrations in supervised training and no demonstrations in reinforcement learning lies an underexplored regime: partial supervision. We ask whether some classes of sequence learning problems become efficiently learnable by exploiting this gap. We address this by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals a partial prefix of the target output. The supervision length is adjusted dynamically for each sample based on the model's past reward signal, allowing it to incrementally learn to complete reasoning chains by conditioning on correct partial solutions. We investigate this intermediate regime between SFT and RL and argue that per-sample curriculum learning is more than a trade-off between efficiency and generality--it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize. Using a synthetic task with latent parity constraints, we show that AdaBack reliably solves problems that are otherwise intractable. On three mathematical reasoning benchmarks, DeepScaleR, MATH, and GSM8k, we find that AdaBack enables models to solve problems that RL alone cannot, acquiring new reasoning capabilities through incremental exposure to partial solutions.

Figures (11)

• Figure 1: AdaBack Update Rule. At epoch $t$, we sample a supervision ratio $\rho^{(i)}_{t} \sim U(\rho_{\min}^{(i)}, \rho_{\max}^{(i)})$ and condition the model's generation on the question and the corresponding partial answer (shown in black text, with unrevealed content grayed out). If the average reward is below a threshold $\tau$, we increase supervision in the next epoch by sampling from the red interval $\rho_{t+1}^{(i)} \sim U(\rho_t^{(i)}, \rho_{\max}^{(i)})$. Otherwise, if the reward exceeds $\tau$, we reduce supervision and sample from $U(0, \rho_t^{(i)})$, the green interval, to make the task harder.
• Figure 2: Training Dynamics.Left: Training and test rewards along with supervision ratios throughout training. With AdaBack, Llama 3.2 1B successfully learns the task in under 700 iterations. Right: Training and test rewards for SFT+RL (red) plateau at 0.1, indicating that only the output format—learned during supervised pretraining—has been retained. Test reward for R3 xi2024training is shown in purple; it reaches only 0.8 reward after more than 16,000 iterations. R3 segments training examples at all whitespace positions and applies RL uniformly over these fragments, resulting in inefficiency due to its non-adaptive strategy.
• Figure 3: AdaBack vs. Standard RL Across Model Initializations. Results from training Llama3-1B on GSM8k dataset. The top row shows results for models initialized with SFT, while the bottom row shows base (non-SFT) models. The left column presents AdaBack training dynamics: train reward increases as supervision ratios (portions) decrease. The right column shows standard RL.
• Figure 4: Pass@k for Llama3-1B SFT-initialized models (left) and base models (right) on GSM8k. AdaBack keeps a significant gap compared to standard RL and improves performance at higher $k$ even without SFT suggesting it expands the solution distribution rather than reweighting known answers (contra yue2025doesreinforcementlearningreally).
• Figure 5: Training reward (left) and test accuracy (right) across different AdaBack reward thresholds. Although there is some difference at the begining of training for high thresholds ($\tau=0.8$ and $\tau=0.9$), the learning curves and performance become indistinguishable as training goes on.