Synthetic Error Injection Fails to Elicit Self-Correction In Language Models

TL;DR

The paper questions whether supervised learning with synthetic error injection can replace reinforcement learning for eliciting self-correction in large language models. It introduces Error Injection Fine-Tuning (EIFT), which inserts erroneous reasoning steps into Chain-of-Thought traces and trains the model to recognize and correct them, while masking the injected error in the loss. Across multiplication and Sudoku tasks on several base models, EIFT yields little to moderate gains and, crucially, fails to generalize error-correcting capabilities to on-policy errors generated by the base models, highlighting a distribution mismatch. The findings suggest that on-policy reinforcement learning remains uniquely effective for self-correction, and any supervised alternative must closely align with the model's native error modes or provide substantially richer supervisory signals. The work emphasizes the importance of error distribution alignment and the solver-verifier dynamics in robust self-correction.

Abstract

Reinforcement learning has become the dominant paradigm for eliciting reasoning and self-correction capabilities in large language models, but its computational expense motivates exploration of alternatives. Inspired by techniques from autonomous driving and robotics, we investigate whether supervised learning with synthetic error injection can induce self-correction abilities in language models. Our approach inserts artificial errors into reasoning chains, masks them, and supervises the model to recognize and correct these mistakes. Despite the intuitive appeal of this method, we find that it fails to significantly improve performance even on simple synthetic tasks across multiple models. Moreover, even when the model catches its own error, it often parrots the original mistake. We find that the distribution shift of synthetic errors to on-policy errors significantly degrades the error-correction capabilities of the fine-tuned model, even with good synthetic coverage of on-policy errors. Our results help explain why on-policy reinforcement learning methods have proven uniquely effective for eliciting self-correction.

Figures (12)

• Figure 1: An illustration of the synthetic error injection process. The (Left) panel shows a correct Chain-of-Thought (CoT) trace used for standard fine-tuning (FT). The (Right) panel details the Error Injection Fine-Tuning (EIFT) methodology, where a correct step from a golden CoT is replaced by a three-part sequence: (1) a synthetically injected erroneous step (e.g., a "Carry Error"), (2) an explicit error recognition step ("Ah! I made a mistake."), and (3) the original correct step as a supervised correction. As indicated by the "Include in Loss?" column, the loss is computed for the recognition and correction steps but masked for the synthetically injected error.
• Figure 2: Types of errors we inject for the multiplication task. We take a correct step, depicted here as the addition step at the top of the figure. We then apply one of six types of errors with various probabilities; see \ref{['tab:error_types']} for more details. The red underlined digits in each box depict the result of applying the corresponding error type.
• Figure 3: An example of a golden Chain-of-Thought (CoT) for the $4 \times 4$ Sudoku task. The solution trajectory is restricted to "naked single" moves, defined as instances where a specific cell has exactly one possible candidate number remaining. The problem instances are generated by solving a random board and removing numbers such that a naked single solution path remains viable.
• Figure 4: Validation of synthetic error distribution alignment with on-policy errors.(a) Error Coverage: The percentage of on-policy model errors that appear at least once within $n=10,000$ samples from our synthetic error injector. We achieve near-perfect coverage ($>95\%$) across all models, indicating the injector correctly identifies the support of the error distribution. (b) Distributional Alignment: The cumulative distribution of the probability assigned by the synthetic injector to the exact on-policy error step (down and to the right is better). For a given probability threshold $p$ on the $x$-axis, the $y$-axis represents what fraction of the 100 errors could be exactly matched with probability at most $p$. The probability mass placed on exact matches (particularly for Qwen2.5 and gemma-3) indicates that the synthetic distribution closely approximates the frequency of natural model errors.
• Figure 5: A comparison of model performance grouped by base model and task. We report the mean accuracy measured on a fixed set of 1000 randomly generated problems, and error bars are 1.96SE. The two model types are (1) FT (Fine-Tuned, red) models which are trained solely on golden CoTs or (2) EIFT (Error-Injection Fine-Tuned, green) on error-injected CoTs. One would expect the EIFT models to have higher accuracy than the FT models. However, across the board, we see only modest ($<5\%$) performance gains for multiplication. For Sudoku tasks the performance is boosted by roughly 10% for Qwen and Llama, but none at all for Gemma.

Theorems & Definitions (6)

• Definition 3.1: FT model
• Definition 3.2: EIFT model
• Definition 3.3: Naked single move
• Definition 3.4: Synthetic errors
• Definition 3.5: FT errors
• Remark 4.1