The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms

TL;DR

The paper argues for a cognitive revolution in interpretability, distinguishing semantic interpretation (latent representations) from algorithmic interpretation (implicit operations) and grounding both in cognitive-science history. It surveys methods from optimization-based, probing, causal probing, and dictionary-learning approaches, analyzing their assumptions, strengths, and limitations, including issues like the Rashomon effect and completeness/selectivity tradeoffs. It highlights circuit discovery as the core of algorithmic interpretation and discusses how causal interventions can test whether discovered circuits actually drive behavior, while recognizing scalability and evaluation challenges. The authors propose moving toward a unified framework that combines semantic and algorithmic perspectives, leveraging causal abstractions to connect representations with operations and aiming for more faithful explanations and safer model deployment.

Abstract

Artificial neural networks have long been understood as "black boxes": though we know their computation graphs and learned parameters, the knowledge encoded by these weights and functions they perform are not inherently interpretable. As such, from the early days of deep learning, there have been efforts to explain these models' behavior and understand them internally; and recently, mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. In this work, we aim to ground MI in the context of cognitive science, which has long struggled with analogous questions in studying and explaining the behavior of "black box" intelligent systems like the human brain. We leverage several important ideas and developments in the history of cognitive science to disentangle divergent objectives in MI and indicate a clear path forward. First, we argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology that shifted the study of human psychology from pure behaviorism toward mental representations and processing. Second, we propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research, semantic interpretation (what latent representations are learned and used) and algorithmic interpretation (what operations are performed over representations) to elucidate their divergent goals and objects of study. Finally, we elaborate the parallels and distinctions between various approaches in both categories, analyze the respective strengths and weaknesses of representative works, clarify underlying assumptions, outline key challenges, and discuss the possibility of unifying these modes of interpretation under a common framework.