Interpreting Language Models Through Concept Descriptions: A Survey

TL;DR

This survey analyzes the nascent field of using generator-based, open-vocabulary concept descriptions to interpret internal components and abstractions of large language models. It distinguishes native model components (neurons, attention heads) from higher-level abstractions (SAEs, circuits), outlining methods to automatically generate descriptions and the datasets and metrics underpinning them. The findings emphasize neuron polysemanticity as a key driver for adopting abstractions like SAEs and circuits, and highlight a maturation in evaluation toward causal and multi-faceted assessments rather than simple correlations. The work maps a roadmap for future research, stressing scalable circuit-level descriptions, cross-domain generalization, introspection of the describers, finer-grained polysemantic models, rigorous causal tests, and standardized benchmarks to enable robust, transparent NLP systems.

Abstract

Understanding the decision-making processes of neural networks is a central goal of mechanistic interpretability. In the context of Large Language Models (LLMs), this involves uncovering the underlying mechanisms and identifying the roles of individual model components such as neurons and attention heads, as well as model abstractions such as the learned sparse features extracted by Sparse Autoencoders (SAEs). A rapidly growing line of work tackles this challenge by using powerful generator models to produce open-vocabulary, natural language concept descriptions for these components. In this paper, we provide the first survey of the emerging field of concept descriptions for model components and abstractions. We chart the key methods for generating these descriptions, the evolving landscape of automated and human metrics for evaluating them, and the datasets that underpin this research. Our synthesis reveals a growing demand for more rigorous, causal evaluation. By outlining the state of the art and identifying key challenges, this survey provides a roadmap for future research toward making models more transparent.

Figures (2)

• Figure 1: Illustration of model components (left column) and model abstractions (right column) of a language model. Components are individual units of a language model, such as neurons or attention heads. Abstractions go beyond the individual components of a model encompassing higher-level representations such as those learned by SAEs, or subgraphs involving multiple components or sparse features, as in circuits. Each component or abstraction can be associated with a human-understandable concept description.
• Figure 2: Overview of descriptions for model components ( neurons, attention heads) and model abstractions ( SAE features, circuits). The top panel shows a schematic example of an automatically generated feature description for a neuron or SAE feature, based on top-activating text samples (same process for both). The middle panel shows an example from neo-2024-interpreting-context-look-ups of how attention head descriptions are generated: first a token-predicting neuron is identified, then prompts that highly activate it are found, the attention heads responsible for its activation are determined, and explanations for these heads are generated. The bottom panel shows a circuit from wang2023interpretability implementing indirect object identification (IOI), where input tokens enter the residual stream and attention heads move information between streams, query/output arrows indicate where they write, key/value arrows where they read, and each class of head has an associated description.