Open Problems in Mechanistic Interpretability

TL;DR

The paper surveys open problems in mechanistic interpretability, spanning methods, foundations, applications, and socio-technical issues. It identifies core methodological challenges (reverse engineering, concept-based probes, and circuit-discovery pipelines) and showcases the need for stronger theory, causal validation, and broader model coverage. It emphasizes practical benefits in monitoring, control, prediction, and scientific insight, while highlighting governance, ethics, and transparency considerations. Overall, progress in mechanistic interpretability could enable safer, more predictable AI and unlock new scientific discoveries, provided it is developed with robust validation, benchmarks, and responsible deployment in mind.

Abstract

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks' capabilities in order to accomplish concrete scientific and engineering goals. Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Figures (6)

• Figure 1: Two approaches to neural network interpretability. (Left) Reverse Engineering is characterized by decomposing networks into functional components and describing how those components interact to produce the network's behavior. It thus aims to 'identify the roles of network components' (\ref{['subsec:reverse-engineering']}). (Right) Concept-based interpretability on the other hand attempts to discover human concepts within neural network internals. It thus aims to 'identify the network components for given roles' (\ref{['subsec:concept-based-interpretability']}).
• Figure 2: The steps of reverse engineering neural networks. (1) Decomposing a network into simpler components. This decomposition might not necessarily use architecturally-defined bases, such as individual neurons or layers (\ref{['subsec:reverse-engineering-step-1']}). (2) Hypothesizing about the functional roles of some or all components (\ref{['subsec:reverse-engineering-step-2']}). (3) Validating whether our hypotheses are correct, creating a cycle in which we iteratively refine our decompositions and hypotheses to improve our understanding of the network (\ref{['subsec:reverse-engineering-step-3']}).
• Figure 3: Three ideas underlying the sparse dictionary learning (SDL) paradigm in mechanistic interpretability. (Left) The linear representation hypothesis states that the map from 'concepts' to neural activations is linear. (Middle) Superposition is the hypothesis that models represent many more concepts than they have dimensions by representing them both sparsely and linearly in activation spaces. (Right) SDL attempts to recover an overcomplete basis of concepts represented in superposition in activation space.
• Figure 4: Sparse dictionary learning has a number of practical and conceptual limitations that cause issues when using it to reverse engineer neural networks (\ref{['par:sdl']}).
• Figure 5: To study the functional role of the blue component numerous approaches are possible. We could study its causes: the purple input components or the red intermediate components via e.g. feature synthesis Olah__2017_FeatureVisualisationolah2020zoom, maximum activating examples Olah__2017_FeatureVisualisationBricken_2023_dictionary, or attributions Sundarajan_2017_IntegratedGradients. Or we could study its effects: the orange output components or the green intermediate components via e.g. the logit lens Nostalgebraist_2020, activation steering turner2024steeringlanguagemodelsactivation, or attributions marks2024sparsefeaturecircuitsdiscovering.