InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning

TL;DR

The paper tackles the credit assignment problem in outcome-reward RL for LLM reasoning by introducing Intervention Training (InT), where the model self-verifies its reasoning against a reference solution, identifies the first error, and proposes a single-step intervention. This intervention, used to create an intervention-guided trace, is then learned via supervised fine-tuning before RL, providing a strong initialization for learning to solve hard problems. Empirical results on challenging math benchmarks (e.g., IMO-AnswerBench) show that InT + RL substantially improves performance, reduces zero-signal rollouts, and often outperforms larger open-source models. The work demonstrates that lightweight, self-generated, step-level interventions can yield high-quality credit signals and more robust reasoning in LLMs, with practical impact for automated mathematical problem solving and beyond.

Abstract

Outcome-reward reinforcement learning (RL) has proven effective at improving the reasoning capabilities of large language models (LLMs). However, standard RL assigns credit only at the level of the final answer, penalizing entire reasoning traces when the outcome is incorrect and uniformly reinforcing all steps when it is correct. As a result, correct intermediate steps may be discouraged in failed traces, while spurious steps may be reinforced in successful ones. We refer to this failure mode as the problem of credit assignment. While a natural remedy is to train a process reward model, accurately optimizing such models to identify corrective reasoning steps remains challenging. We introduce Intervention Training (InT), a training paradigm in which the model performs fine-grained credit assignment on its own reasoning traces by proposing short, targeted corrections that steer trajectories toward higher reward. Using reference solutions commonly available in mathematical reasoning datasets and exploiting the fact that verifying a model-generated solution is easier than generating a correct one from scratch, the model identifies the first error in its reasoning and proposes a single-step intervention to redirect the trajectory toward the correct solution. We then apply supervised fine-tuning (SFT) to the on-policy rollout up to the point of error concatenated with the intervention, localizing error to the specific step that caused failure. We show that the resulting model serves as a far better initialization for RL training. After running InT and subsequent fine-tuning with RL, we improve accuracy by nearly 14% over a 4B-parameter base model on IMO-AnswerBench, outperforming larger open-source models such as gpt-oss-20b.

Figures (21)

• Figure 1: Intervention training (InT) for improving credit assignment.InT proposes single-step interventions to replace incorrect intermediate steps in reasoning traces (1). Conditioned on these localized corrections, the model can generate counterfactual continuations that succeed where the original failed (2). We then perform supervised fine-tuning on these interventions, enabling effective credit assignment by upweighting the likelihood of the interventions in place of the mistakes.
• Figure 2: Location of first occurring mistakes in incorrect trajectories. Incorrect rollouts often do not begin with mistakes and may contain a significant number of correct preceding steps/tokens. Detected by Gemini 2.5 Pro with Prompt \ref{['box:gemini-prompt']}.
• Figure 3: Example mistakes encountered by Qwen3-4B-Instruct and proposed interventions. By correcting the mistakes with interventions, the rollouts lead to correct final answers as opposed to originally incorrect ones.
• Figure 4: Performance comparison of interventions. Continuing the rollout by concatenating the proposed intervention step $\tilde{\mathbf{y}}_{t^*}$ to the prefix $\mathbf{y}_{<t^*}$ yields substantially higher accuracy (left) and a larger number of unique problems solved (right) than either continuing with the original erroneous step $\mathbf{y}_{t^*}$ or continuing directly from the prefix without any intervention ($-$).
• Figure 5: Intervention training (InT).InT verifies incorrect rollouts against reference solutions to identify the step $t^*$ at which the first error ${\mathbf{y}}_{t^*})$ occurs, proposes alternative step $\tilde{\mathbf{y}}_{t^*})$, and performs SFT on these steps before RL.