How Does Unfaithful Reasoning Emerge from Autoregressive Training? A Study of Synthetic Experiments

TL;DR

This study formalizes faithful CoT reasoning and its emergence under autoregressive training using the Arithmetic Expression Reasoning (AER) synthetic task. By defining consistency-based and intervention-based faithfulness, it reveals a critical noise threshold below which faithful, stepwise reasoning is possible and above which unfaithful skip-step behaviors emerge, driven by simplicity bias. The authors identify four training-phase regimes, show that mixed reasoning temporarily increases prediction entropy, and demonstrate that models develop internal uncertainty and self-verification signals via mechanistic analysis. These findings illuminate the underpinnings of CoT unfaithfulness and offer a principled framework for evaluating and mitigating unreliable reasoning in LLMs, with implications for safety and alignment.

Abstract

Chain-of-thought (CoT) reasoning generated by large language models (LLMs) is often unfaithful: intermediate steps can be logically inconsistent or fail to reflect the causal relationship leading to the final answer. Despite extensive empirical observations, a fundamental understanding of CoT is lacking--what constitutes faithful CoT reasoning, and how unfaithfulness emerges from autoregressive training. We study these questions using well-controlled synthetic experiments, training small transformers on noisy data to solve modular arithmetic expressions step by step, a task we term Arithmetic Expression Reasoning. We find that models can learn faithful reasoning that causally follows the underlying arithmetic rules, but only when the training noise is below a critical threshold, a phenomenon attributable to simplicity bias. At higher noise levels, training dynamics exhibit a transition from faithful stepwise reasoning to unfaithful skip-step reasoning via an intermediate mixed mode characterized by a transient increase in prediction entropy. Mechanistic analysis reveals that models learn to encode internal uncertainty by resolving inconsistent reasoning steps, which suggests the emergence of implicit self-verification from autoregressive training.

Figures (23)

• Figure 1: Illustration of our Arithmetic Expression Reasoning (AER) task for CoT reasoning. An input sequence of the format (\ref{['eq:format']}) is sampled and tokenized. Then a small transformer is trained from scratch on such data in the standard autoregressive fashion.
• Figure 2: Training small transformers for the AER task. Left/Middle: evaluating CoT reasoning under two faithfulness definitions. We train multiple transformers separately on datasets of varying noise levels ($\varepsilon_1$: prompt noise, $\varepsilon_2$: reasoning noise). Both unfaithfulness metrics are small at low noise (faithful) until critical thresholds of $\varepsilon_2$, beyond which unfaithfulness sharply increases---orange curves show generated reasoning chains become less consistent, and blue curves show interventions on reasoning barely change generated solutions. Right: Prediction entropy across training. The model experiences four phases: format following (P0), stepwise reasoning (P1), mixed reasoning (P2), and skip-step reasoning (P3). Its prediction uncertainty temporarily increases in P2 as it resolves conflicting information in the reasoning chain, which suggests implicit self-verification behavior.
• Figure 3: Larger complexity gap between $e_1$ and $e_2$ reduces unfaithfulness. Fixing $\varepsilon_1=0.01$, we sweep $\varepsilon_2$ and compare training transformers on two-operator prompts $e_1$ and three-operator prompts. Both metrics indicate that larger difficulty gaps encourage stepwise reasoning and reduce unfaithfulness.
• Figure 4: Training dynamics exhibit distinct phases of reasoning modes. We conduct three experiments under different noise configurations and evaluate models with three metrics (logarithmic x-axis). Left: Noise levels $(\varepsilon_1, \varepsilon_2) = (0.01, 0.1)$. The model exhibits four distinct reasoning phases, validating our proposed phase characterization. Middle: $(\varepsilon_1, \varepsilon_2) = (0.1, 0.1)$. The model exhibits only the first three phases, as equal noise levels incentivizes the model to stay in the mixed reasoning mode P2. Right: $(\varepsilon_1, \varepsilon_2) = (0.1, 0.01)$. The model exhibits only the first three phases, as lower prompt noise incentivizes stepwise reasoning P1.
• Figure 5: Prediction entropy shows a temporary ascent despite continuously decreasing training loss. Models encode reasoning uncertainty in the mixed reasoning mode, which matches most sharp descents of the loss.

Theorems & Definitions (1)

• Definition 2.1