Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i

Kola Ayonrinde, Louis Jaburi

TL;DR

The paper reframes neural network interpretability as a rigorous science, proposing the Explanatory View in which internal mechanisms provide causal, model-level explanations whose fidelity is measured by explanatory faithfulness. It formalizes Mechanistic Interpretability (MI) as producing Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations, and argues that generalisation reflects compressible structure that ur-explanations capture via representations. A central conjecture, the Principle of Explanatory Optimism, claims that the algorithmic generalisation of neural networks is human-understandable, justifying MI's pursuit despite value- and theory-ladenness. The work also delineates the limits of MI, especially in system-level contexts and low-abstraction explanations, and sketches a research agenda to formalize and test EO while emphasizing the role of explanatory virtues and human-computer interaction. Overall, the paper advocates for a realism-based, explanatory foundation for interpretability with potential implications for AI safety, ethics, and cognitive science.

Abstract

Mechanistic Interpretability aims to understand neural networks through causal explanations. We argue for the Explanatory View Hypothesis: that Mechanistic Interpretability research is a principled approach to understanding models because neural networks contain implicit explanations which can be extracted and understood. We hence show that Explanatory Faithfulness, an assessment of how well an explanation fits a model, is well-defined. We propose a definition of Mechanistic Interpretability (MI) as the practice of producing Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations of neural networks, allowing us to distinguish MI from other interpretability paradigms and detail MI's inherent limits. We formulate the Principle of Explanatory Optimism, a conjecture which we argue is a necessary precondition for the success of Mechanistic Interpretability.

A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i

TL;DR

The paper reframes neural network interpretability as a rigorous science, proposing the Explanatory View in which internal mechanisms provide causal, model-level explanations whose fidelity is measured by explanatory faithfulness. It formalizes Mechanistic Interpretability (MI) as producing Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations, and argues that generalisation reflects compressible structure that ur-explanations capture via representations. A central conjecture, the Principle of Explanatory Optimism, claims that the algorithmic generalisation of neural networks is human-understandable, justifying MI's pursuit despite value- and theory-ladenness. The work also delineates the limits of MI, especially in system-level contexts and low-abstraction explanations, and sketches a research agenda to formalize and test EO while emphasizing the role of explanatory virtues and human-computer interaction. Overall, the paper advocates for a realism-based, explanatory foundation for interpretability with potential implications for AI safety, ethics, and cognitive science.

Abstract

Mechanistic Interpretability aims to understand neural networks through causal explanations. We argue for the Explanatory View Hypothesis: that Mechanistic Interpretability research is a principled approach to understanding models because neural networks contain implicit explanations which can be extracted and understood. We hence show that Explanatory Faithfulness, an assessment of how well an explanation fits a model, is well-defined. We propose a definition of Mechanistic Interpretability (MI) as the practice of producing Model-level, Ontic, Causal-Mechanistic, and Falsifiable explanations of neural networks, allowing us to distinguish MI from other interpretability paradigms and detail MI's inherent limits. We formulate the Principle of Explanatory Optimism, a conjecture which we argue is a necessary precondition for the success of Mechanistic Interpretability.
Paper Structure (52 sections, 2 figures)

This paper contains 52 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Series Structure
  4. The Strange Science
  5. Paper Structure.
  6. Explanations and Interpretability
  7. Scientific Explanation
  8. Understanding and Compression.
  9. From Induction To Explanation
  10. Generalisation: What We're Explaining When We're Explaining Neural Networks
  11. Explanatory Faithfulness
  12. Neural Networks Perform Computations over Representations
  13. Computation.
  14. Representation.
  15. The Goal of Interpretability
  16. ...and 37 more sections

Figures (2)

  • Figure 1: A Venn diagram showing the relationship between the concept spaces of the machine M and human interpreter H. The machine and human have some shared concepts which they can use to communicate ($\textcolor{blue}{C_M} \cap \textcolor{red}{C_H}$) but there are many concepts that the machine uses that the human does not understand ($\textcolor{blue}{C_M} \setminus \textcolor{red}{C_H}$). The set of Alien Concepts$\textcolor{green!60!black}{C_A} \subset (\textcolor{blue}{C_M} \setminus \textcolor{red}{C_H})$, is a subset of the Machine-concepts. Alien Concepts are causally relevant for the model's computation but are fundamentally incomprehensible to humans. If this set is large or important, then Interpretability may be highly limited.
  • Figure 2: When studying InceptionV1 szegedy2015inceptionv1, olah2020zoom_in_circuits note that though the network was trained only to classify ImageNet images russakovsky2015imagenet, the network learned intermediate representations that were useful for the image classification task. For example, in order to detect cars, the network internally learned concepts for windows, wheels and car bodies. This is an example of Conceptual Engineering in neural networks. [Image from olah2020zoom_in_circuits]