Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii

TL;DR

The paper introduces the Explanatory Virtues Framework to assess Mechanistic Interpretability explanations through a pluralist lens drawn from Bayesian, Kuhnian, Deutschian, and Nomological accounts. It formalizes validity for MI explanations as Model-level, Ontic, Causal-Mechanistic, and Falsifiable, and then defines a suite of theoretical virtues (e.g., Accuracy, Precision, Descriptiveness, Power, Unification, Falsifiability, Hard-to-Vary) and empirical virtues (e.g., Mundane Accuracy, Descriptiveness, Co-Explanation, Fruitfulness). The authors apply the framework to MI methods (Clustering, Sparse Autoencoders, Causal Abstraction, and Compact Proofs), analyze how well each embodies the virtues, and propose prioritized directions—Simplicity/Compression, Unification/Co-Explanation, and Nomological Principles—for advancing reliable explanations. They illustrate how Compact Proofs can tightly couple explanatory quality with verifiable guarantees and argue for nomological theories to unify diverse MI observations. Overall, the work provides a principled, multi-criteria approach to evaluating explanations, with practical implications for monitoring, predicting, and steering neural systems in AI safety, ethics, and cognitive science contexts.

Abstract

Mechanistic Interpretability (MI) aims to understand neural networks through causal explanations. Though MI has many explanation-generating methods, progress has been limited by the lack of a universal approach to evaluating explanations. Here we analyse the fundamental question "What makes a good explanation?" We introduce a pluralist Explanatory Virtues Framework drawing on four perspectives from the Philosophy of Science - the Bayesian, Kuhnian, Deutschian, and Nomological - to systematically evaluate and improve explanations in MI. We find that Compact Proofs consider many explanatory virtues and are hence a promising approach. Fruitful research directions implied by our framework include (1) clearly defining explanatory simplicity, (2) focusing on unifying explanations and (3) deriving universal principles for neural networks. Improved MI methods enhance our ability to monitor, predict, and steer AI systems.

Figures (5)

• Figure 1: A Directed Acyclic Graph representation of the Explanatory Virtues Framework showing the relationships between virtues. Empirical virtues are coloured orange and theoretical virtues are coloured blue. We show the virtues which directly depend on each other with bold arrows ($\boldsymbol{\rightarrow}$) and those which are highly related with dashed arrows ($\dashrightarrow$). The Explanatory Virtues which are essential for any scientific explanation (Falsifiability and Causal-Mechanisticity) to be valid are denoted with an exclamation mark; the most important virtues to decide between explanations (Simplicity, Hard-to-Varyness, and Fruitfulness) are marked with a star. \ref{['sec:values_rubric']} details a rubric for assessing explanatory methods. \ref{['sec:straightforward_explanations']} provides an example illustrating the importance of Simplicity as an explanatory virtue.
• Figure 2: Given some (possibly intermediate) embeddings ($\mathbf{x}$), a clustering explanation can be produced by assigning $\mathbf{x}$ to a cluster $C_i$, where the n clusters partition the input space into disjoint regions. Here $C_1 \cup C_2 \cup \ldots \cup C_n = \mathbb{R}^N$ and $C_i \cap C_j = \emptyset$$\forall i \neq j$. The explanation is then given by taking the behaviour of the model on some cluster representative, or centroid, $\mu_i \in C_i$. We can intuitively see this as performing a quotient operation on the input space, where the model behaviour is approximated by a piecewise constant function. [Image from google_clustering].
• Figure 3: (a) The SAE architecture. An encoder provides some set of latents (or feature activations) in the feature basis. We have some decoder map, Dec, which is a linear combination of the columns of the feature dictionary weighted by the sparse latents. We say informally that these latents correspond to the input activations if, under the decoder map, Dec. (b) If $\mathbf{x}$ and $\mathbf{z}$ correspond in the above sense then the natural language explanation of the input activations $\mathbf{x}$ is given as $e(\mathbf{x}) = e'(\mathbf{z})$; that is the explanation of the latents using the automated interpretability process $e'(\mathbf{z})$paulo2024autointerpsaebench2024bills2023autointerpkayonrinde2025incoherent_saes. We can measure the mathematical description length (Conciseness) of the explanation $e(\mathbf{x})$ as the number of bits required to describe the latents $\mathbf{z}$ayonrinde2024_mdl_saes. [Images from ayonrinde2024_mdl_saeskayonrinde2025incoherent_saes]
• Figure 4: A circuit explanation is a Causal-Mechanistic explanation such that the circuit C is a constructive abstraction of a neural network's computational graph M if there exists a partition the variables in M such that each high-level variables in C correspond to a low-level partition cell in M and interventions on M correspond to interventions on C. For example in \ref{['fig:circuits_acdc']}Leftconmy2023acdc, the IOI circuit wang_ioi (highlighted in red) is recovered from the computational graph of GPT-2 Small. [Image from conmy2023acdc].
• Figure 5: (a) Compact Proofs evaluate explanations on two metrics, their compactness (FLOPs to Verify Proof) and their accuracy (Model Performance Lower Bound). These two metrics can be assessed on a Pareto frontier. (b) A good explanation should push the frontier towards the upper left corner (i.e. more accurate and compact proofs). [Image from gross2024compact.]