Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision

TL;DR

This work identifies a novel phenomenon: under outcome-only supervision, out-of-distribution (OOD) performance can continue improving as training-time reasoning length increases, even after in-distribution performance has saturated, suggesting that robustness may require a larger budget than ID validation alone would indicate.

Abstract

Training LLMs to think and reason for longer has become a key ingredient in building state-of-the-art models that can solve complex problems previously out of reach. Recent efforts pursue this in different ways, such as RL fine-tuning to elicit long CoT or scaling latent reasoning through architectural recurrence. This makes reasoning length an important scaling knob. In this work, we identify a novel phenomenon (both theoretically and experimentally): under outcome-only supervision, out-of-distribution (OOD) performance can continue improving as training-time reasoning length (e.g., the token budget in RL, or the loop count in looped Transformers) increases, even after in-distribution (ID) performance has saturated. This suggests that robustness may require a larger budget than ID validation alone would indicate. We provide theoretical explanations via two mechanisms: (i) self-iteration can induce a stronger inductive bias in the hypothesis class, reshaping ID-optimal solutions in ways that improve OOD generalization; and (ii) when shortcut solutions that work for ID samples but not for OOD samples persist in the hypothesis class, regularization can reduce the learned solution's reliance on these shortcuts as the number of self-iterations increases. We complement the theory with empirical evidence from two realizations of scaling training-time reasoning length: increasing the number of loops in looped Transformers on a synthetic task, and increasing token budgets during RL fine-tuning of LLMs on mathematical reasoning.

Figures (5)

• Figure 1: Looped Transformers trained on $4$-hop problems. Results averaged over 8 random seeds, with shaded regions showing std. Increasing the loop count yields OOD gains up to around 44 loops, while ID accuracy saturates near $100\%$ by around 2 loops.
• Figure 2: RL fine-tuning of Qwen2.5-1.5B-Instruct on math data with varying training-time token budgets. Results are averaged over 3 random seeds. ID accuracy peaks at 256 and slightly drops thereafter, while OOD accuracy steadily improves throughout. This trend persists even when fixing the evaluation-time budget to 4096 (b), indicating that the effect is truly driven by the training-time budget.
• Figure 3: For small budgets (128/256), the mean reward (left) remains near 0 early in training and rises sharply when the mean response length (right) abruptly drops, indicating a transition to shorter, less step-by-step outputs. In contrast, with a large budget (4096), response length increases throughout training.
• Figure 4: Results for looped Transformers on the $4$-hop task with loop embeddings show a pattern consistent with Figure \ref{['fig:phop']}. ID accuracy saturates around 2 loops, while OOD accuracy continues improving up to around 36 loops.
• Figure 5: Pass@1 results show the same qualitative pattern: ID performance oscillates slightly more but exhibits no clear improvement beyond a token budget of 256, while OOD accuracy continues to improve steadily under both evaluation setups.

Theorems & Definitions (27)

• Proposition 3.1
• proof
• Theorem 3.2
• Theorem 3.3
• Theorem 3.5
• Proposition 3.6
• Lemma 1.1: Large iterates collapse height
• proof
• Lemma 1.2: Only two interpolants can have height at most $1$
• proof