Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neuroscience

TL;DR

The paper addresses the lack of a unifying framework for AI inner interpretability and notes critiques about usefulness and generalization.It proposes a multilevel interpretability framework inspired by Cognitive Neuroscience, spanning computational, algorithmic, primitive representations, and implementation levels.Concrete strategies include formal computational problems (e.g., $F_I=(S,R,\bot)$ and $F=(S,R,A)\in\mathcal{D}$), algorithmic descriptions, primitive representations such as key-value memory, rigorous severe tests, and invariance analysis.The framework aims to improve rigor, comparability, safety, and scalability of mechanistic explanations and to harmonize past research with future directions.

Abstract

Inner Interpretability is a promising emerging field tasked with uncovering the inner mechanisms of AI systems, though how to develop these mechanistic theories is still much debated. Moreover, recent critiques raise issues that question its usefulness to advance the broader goals of AI. However, it has been overlooked that these issues resemble those that have been grappled with in another field: Cognitive Neuroscience. Here we draw the relevant connections and highlight lessons that can be transferred productively between fields. Based on these, we propose a general conceptual framework and give concrete methodological strategies for building mechanistic explanations in AI inner interpretability research. With this conceptual framework, Inner Interpretability can fend off critiques and position itself on a productive path to explain AI systems.

Figures (3)

• Figure 1: The fields of Inner Interpretability and Cognitive Neuroscience aim to mechanistically explain the behavior of artificial and biological systems, respectively. The multilevel explanatory framework proposed here draws out the parallels and suggests strategies that can be transferred between fields to tackle current issues in Inner Interpretability (shown in red).
• Figure 2: Examples of the primitives and implementation levels using the key-value memory pairs system in MLP layers.
• Figure 3: Schematic of the key techniques deployed by different disciplines in Cognitive Neuroscience organized according to whether they promote top-down or bottom-up approaches to the discovery of mechanistic explanations. Note that only radical (top-down/bottom-up) approaches propose to reach inner levels through a one-directional use of these techniques. In practice, the discovery of mechanistic explanations involves a healthy combination of techniques from top-down and bottom-up approaches.

Theorems & Definitions (2)

• Definition 4.1: Facts and fact domains
• Definition 4.2: Factual recall