Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networks

TL;DR

The paper examines how interpreting neural networks via counterfactual interventions faces two core problems: overdetermination, where multiple independent causes can suffice for an effect, and non-transitivity, where counterfactual dependencies do not propagate cleanly through intermediate nodes. It presents a framework for analyzing these issues through causal theory, a case study on Pythia 70M, and a set of practical recommendations. The authors argue that current methods may systematically miss redundant causes and misinterpret causal graphs unless interventions consider sets of components and retain some local dependencies. They also propose exploring alternative causal theories and better causal mediators to improve recall, precision, and interpretability in mechanistic analyses.

Abstract

Interpretability research takes counterfactual theories of causality for granted. Most causal methods rely on counterfactual interventions to inputs or the activations of particular model components, followed by observations of the change in models' output logits or behaviors. While this yields more faithful evidence than correlational methods, counterfactuals nonetheless have key problems that bias our findings in specific and predictable ways. Specifically, (i) counterfactual theories do not effectively capture multiple independently sufficient causes of the same effect, which leads us to miss certain causes entirely; and (ii) counterfactual dependencies in neural networks are generally not transitive, which complicates methods for extracting and interpreting causal graphs from neural networks. We discuss the implications of these challenges for interpretability researchers and propose concrete suggestions for future work.

Figures (3)

• Figure 1: Illustration of the non-transitivity of counterfactual dependence. If an output $y$ depends on $B$, and if $B$ depends on $A$ (potentially in addition to other nodes), then it does not necessarily entail that $y$ depends on $A$.
• Figure 2: Sparse feature circuit for layer 3 attention feature 14579, a general succession feature (denoted y in the figure). Best viewed zoomed in. Darker blue shades indicate higher-magnitude positive effects, and darker red shades indicate higher-magnitude negative effects. Given inputs such as "1, 2, 3,", the model's first layers detect each number and the punctuation between them (a). The number detectors are inputs to number list detectors and increasing number sequence detectors (b). These features then inform induction features (c) that are only active in specific contexts or on particular classes of phenomena (like number sequences). These aid more general incrementor/succession features (d).
• Figure 3: Illustration of the result of intervening on individual model components. The captions depict the actual causal graph, while the visualizations depict the causal graph that would have been discovered via ablations to individual components. Nodes in blue are in the actual causal graph and would be discoverable. Nodes in pink are in the actual causal graph, but would not have been discovered. Black edges are cause-effect relationships, whereas red edges indicate preemption.