Enhancing Automated Interpretability with Output-Centric Feature Descriptions

TL;DR

The paper argues that faithful LLM feature descriptions must account for both what activates a feature and how that activation affects outputs, challenging input-only approaches. It introduces two output-centric methods, VocabProj and TokenChange, and demonstrates that while MaxAct excels at identifying activating inputs, the proposed methods better capture causal output effects; ensembles that combine all methods yield the strongest overall descriptions. Extensive experiments across SAE and neuron features in Gemma-2, Llama-3.1, and GPT-2 small show that output-centric approaches are computationally efficient and, when used with ensembles, robust across models and layers. The work enables not only more faithful interpretability pipelines but also practical gains in steering and reviving dead features, with code and descriptions publicly available for replication and extension.

Abstract

Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. These descriptions are derived using inputs that activate the feature, which may be a dimension or a direction in the model's representation space. However, identifying activating inputs is costly, and the mechanistic role of a feature in model behavior is determined both by how inputs cause a feature to activate and by how feature activation affects outputs. Using steering evaluations, we reveal that current pipelines provide descriptions that fail to capture the causal effect of the feature on outputs. To fix this, we propose efficient, output-centric methods for automatically generating feature descriptions. These methods use the tokens weighted higher after feature stimulation or the highest weight tokens after applying the vocabulary "unembedding" head directly to the feature. Our output-centric descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions, but combining the two leads to the best performance on both input and output evaluations. Lastly, we show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead".

Figures (17)

• Figure 1: We posit that a faithful description of a feature should consider both model inputs that activate it (left, marked words cause the highest activations) and the effect it introduces to the model's outputs (right).
• Figure 2: Illustration of our feature description evaluation, considering the description's faithfulness with respect to both the input (middle panel) and output (lower panel) of the model.
• Figure 3: Performance of the various methods on the proposed metrics, for Gemma-2 2B (upper row), Llama-3.1 8B (lower left), and Llama-3.1 8B Instruct (lower right). For the output metric, the baseline (dashed black line) is $1/3$ since the judge LLM picks between three sets of texts.
• Figure 4: Prompt given to the judge LLM for the input-based evaluation.
• Figure 5: Human evaluation results for 100 features across three methods, for input- and output-faithfulness.