Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning

TL;DR

The paper addresses the stagnation of RLVR training on saturated problems where rewards become nearly deterministic, hindering learning signals. It introduces failure-prefix conditioning, which starts training from prefixes of incorrect trajectories to expose failure-prone states and recover informative failures. Empirical results on math-reasoning benchmarks show gains comparable to medium-difficulty data while preserving token efficiency, and the method enhances robustness to misleading prefixes. An iterative prefix-refresh procedure further improves performance, suggesting saturated problems remain valuable for training when exploration targets failure modes rather than correct solutions.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved the reasoning abilities of large language models (LLMs), yet training often stalls as problems become saturated. We identify the core challenge as the poor accessibility of informative failures: learning signals exist but are rarely encountered during standard rollouts. To address this, we propose failure-prefix conditioning, a simple and effective method for learning from saturated problems. Rather than starting from the original question, our approach reallocates exploration by conditioning training on prefixes derived from rare incorrect reasoning trajectories, thereby exposing the model to failure-prone states. We observe that failure-prefix conditioning yields performance gains matching those of training on medium-difficulty problems, while preserving token efficiency. Furthermore, we analyze the model's robustness, finding that our method reduces performance degradation under misleading failure prefixes, albeit with a mild trade-off in adherence to correct early reasoning. Finally, we demonstrate that an iterative approach, which refreshes failure prefixes during training, unlocks additional gains after performance plateaus. Overall, our results suggest that failure-prefix conditioning offers an effective pathway to extend RLVR training on saturated problems.

Figures (6)

• Figure 1: Illustration of standard GRPO training and failure-prefix conditioning on saturated problems. While standard GRPO predominantly generates correct rollouts, failure-prefix conditioning exposes the model to failure-prone reasoning states, making informative failures more accessible.
• Figure 2: Performance comparison across models. Left: Mean Pass@$k$ values. Middle: Average token count of responses across benchmarks. Right: Mean accuracy under different inference token limits. Failure-prefix conditioning consistently improves performance while maintaining token efficiency.
• Figure 3: Ablation study on target accuracy $\tau$. We plot the mean accuracy across benchmarks for $\tau \in \{0.25, 0.50, 0.75\}$ over 800 gradient steps. Peak performance points are highlighted for each setting.
• Figure 4: Rollout accuracy versus prefix length (% of trajectory) when conditioning on correct and incorrect prefixes.
• Figure 5: Effect of iterative failure-prefix conditioning on training dynamics. Left: Training reward curves for prolonged iteration 1 (steps 0–800) and iteration 2 that forks at step 400 and proceeds with updated failure prefixes through step 800. Right: Mean accuracy across benchmarks for both iterations measured from steps 400–800, with peak performance point for each model highlighted.