Automatically Interpreting Millions of Features in Large Language Models

TL;DR

This work tackles the challenge of interpreting millions of SAE features in large language models by introducing an automated NL-explanation pipeline. It replaces costly simulation with five cheap, complementary scoring methods (Detection, Fuzzing, Surprisal, Embedding, Intervention) and demonstrates that SAE latents are more interpretable than neurons across multiple configurations. The authors provide large-scale evidence, practical guidelines, and a public codebase for generating and evaluating explanations, enabling scalable mechanistic insights and potential model steering. The results underscore the value of SAE-based representations for interpretability and lay groundwork for broader adoption across LLM research and tooling.

Abstract

While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which may be more easily interpretable. However, these SAEs can have millions of distinct latent features, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of explanations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a feature, which we find explains features that are not recalled by existing methods. We propose guidelines for generating better explanations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. We use our explanations to measure the semantic similarity of independently trained SAEs, and find that SAEs trained on nearby layers of the residual stream are highly similar. Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-$k$ postprocessing. Our code is available at https://github.com/EleutherAI/sae-auto-interp, and our explanations are available at https://huggingface.co/datasets/EleutherAI/auto_interp_explanations.

Figures (10)

• Figure 1: Auto-interpretability pipeline. The first step of the interpretability pipeline is collecting the activations of the target SAE over a broad range of text. In this figure we show short contexts for legibility, and over represent the active feature, which in reality would be active in a very small portion of the text. The activating contexts are selected and shown to an explainer LLM, prompt in Appendix \ref{['app:explainer-prompt']}, which provides a short interpretation. This interpretation is then given to a scorer LLM that is tasked to use this interpretation to distinguish activating from non activating contexts, see Section \ref{['sec:scoring_methods']}
• Figure 2: Fuzzing and detection scores for different sampling techniques. Panels a) and b) show the distributions of fuzzing and detection scores, respectively, as a function of different example sampling methods for interpretation generation. Sampling only from the top activation gets on average lower accuracy in fuzzing and on detection when compared with uniform sampling and stratified sampling. The distributions from random sampling and sampling from quantiles are very similar. Panels c) and d) measure how the interpretations generalize across activation quantiles, showing that interpretations generated from the top quantiles are better at distinguishing non-activating examples, but have lower accuracy on other quantiles, especially on the lower activating deciles. This also happens for the other interpretations, but the accuracy does not drop as much in lower activating deciles. We also show the scores of random explanations.
• Figure 3: Intervention scores. Here we present intervention scores (Sec \ref{['sec:output-scoring']}) for SAE features in Gemma 2 9B at layer 32. Left: SAE features are more interpretable than random features, especially when intervening more strongly. Our explainer also produces interpretations that are scored higher than random interpretations. Right: Many features that would normally be uninterpreted when using context-based automatic interpretability are interpretable in terms of their effects on output.
• Figure A1: SAE features interpretations for a random sentence. To visualize the feature interpretations produced, we select a sentence taken from the RPJv2 dataset. We selected 4 tokens in different positions in that sentence and filter for features that are active in different layers. Then we randomly select active features and their corresponding interpretations to display. We display the detection and fuzzing scores of each interpretation, which indicate how well it explains other examples in the dataset (see Section 3 for details on these scores). The features selected had high activation, but were not cherry-picked based on interpretations or scores.
• Figure A2: Activating contexts of feature 209 (top) and 293 (bottom), from layers 8 and 32, respectively, of the 131k feature SAE trained on the residual stream of Gemma 2 9b. The shown examples are similar to the ones given to the explainer model to come up with interpretations.