Thought Branches: Interpreting LLM Reasoning Requires Resampling

TL;DR

This work reframes LLM reasoning as a distribution over many CoTs, arguing that single-sample analyses miss causal structure. It introduces resampling-based methods, including resilience and counterfactual++ metrics, to quantify how individual reasoning steps influence downstream decisions and to distinguish transient from persistent content. The authors demonstrate on-policy resampling as a principled way to effect causal changes, showing larger and more reliable effects than off-policy edits, and reveal that self-preservation cues often have minimal causal impact. They also adapt causal mediation analysis to study unfaithful CoTs, revealing subtle, cumulative influences from hints on final outcomes. Overall, the approach enables clearer narratives of model reasoning and principled CoT interventions with potential to improve safety and interpretability in real-world AI systems.

Abstract

Most work interpreting reasoning models studies only a single chain-of-thought (CoT), yet these models define distributions over many possible CoTs. We argue that studying a single sample is inadequate for understanding causal influence and the underlying computation. Though fully specifying this distribution is intractable, it can be understood by sampling. We present case studies using resampling to investigate model decisions. First, when a model states a reason for its action, does that reason actually cause the action? In "agentic misalignment" scenarios, we resample specific sentences to measure their downstream effects. Self-preservation sentences have small causal impact, suggesting they do not meaningfully drive blackmail. Second, are artificial edits to CoT sufficient for steering reasoning? These are common in literature, yet take the model off-policy. Resampling and selecting a completion with the desired property is a principled on-policy alternative. We find off-policy interventions yield small and unstable effects compared to resampling in decision-making tasks. Third, how do we understand the effect of removing a reasoning step when the model may repeat it post-edit? We introduce a resilience metric that repeatedly resamples to prevent similar content from reappearing downstream. Critical planning statements resist removal but have large effects when eliminated. Fourth, since CoT is sometimes "unfaithful", can our methods teach us anything in these settings? Adapting causal mediation analysis, we find that hints that have a causal effect on the output without being explicitly mentioned exert a subtle and cumulative influence on the CoT that persists even if the hint is removed. Overall, studying distributions via resampling enables reliable causal analysis, clearer narratives of model reasoning, and principled CoT interventions.

Figures (12)

• Figure 1: Resilience score of sentence categories across four reasoning models. Self-preservation consistently shows the lowest resilience across all models, i.e., it is the easiest sentence category to get rid of in the CoTs.
• Figure 2: Counterfactual++ importance metric of sentence categories across four reasoning models. Derived from the resilience-based resampling procedure. Shows low overall causal effect for self-preservation.
• Figure 3: Sentence insertions versus outcome change across blackmail and whistleblow scenarios. Off-policy edits cluster near zero effect. On-policy resampled interventions achieve larger and more directional effects.
• Figure 4: Each colored line represents a single unfaithful CoT from one of ten MMLU problems; only ten are shown to keep the visualization clear. The y-axis shows the percentage of resampled rollouts that produced the hinted answer after transplanting the CoT up to the sentence position on the x-axis. The box plots aggregate this data across 40 problems, showing the median and interquartile range.
• Figure 5: On the left, we summarize the prompt: short system prompt ( 50 tokens), brief description of the job ( 20 tokens), candidate name and email ( 10 tokens), and resume ( 1150 tokens). In the middle, we show example statements exerting strong causal effects (\ref{['appendix-resume-sentences']}). On the right, each scatterplot dot represents one sentence cluster, showing its average causal effect (x-dimension) and differences in likelihoods to be mentioned for a Black female (positive) or White male (negative) candidate. Clustering was performed independently for the eight resumes, and each dot has been arbitrarily colored based on its resume.