Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders

TL;DR

This work investigates whether interpretability features extracted from a Transformer’s internal feed-forward (FF) layers—viewed as key–value memories (FF-KV)—offer interpretability benefits comparable to proxy-based approaches like Sparse Autoencoders (SAEs) and Transcoders. Using SAEBench-based automatic metrics and human evaluations, the authors compare vanilla FF-KV, TopK and normalized FF-KV, SAEs, and Transcoders across multiple models, finding that FF-KV features achieve interpretability scores in a similar range to SAEs, with FF-KV sometimes outperforming SAEs in absorption and feature independence. A faithfulness analysis using Transcoder as a proxy reveals that many Transcoder features do not align with FF features, suggesting proxies may hallucinate new features rather than faithfully translating the original FF mechanisms. The study concludes that SAEs and Transcoders offer limited advantages over direct FF-KV analysis and argues for FF-KV features to serve as a strong baseline when evaluating feature-discovery interpretability methods, while highlighting the need for interpretability-by-design in future models.

Abstract

Recent interpretability work on large language models (LLMs) has been increasingly dominated by a feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated. This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately, only a few studies have made such comparisons systematically so far. In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAEs and FFs diverged. These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors, and FF key-value parameters serve as a strong baseline in modern interpretability research.

Figures (42)

• Figure 1: Evaluation scores for TopK FF-KVs at Layer 13 on Gemma-2-2B, under a different sparsity value $k$. A higher $k$ indicates a higher sparsity. Shaded areas denote $\pm 2$ standard errors of the mean, computed across multiple seeds and evaluation settings.
• Figure 2: Relationship between FF hidden dimension size (model scale) and RAVEL scores.
• Figure 3: Histogram of aligned features numbers distribution for each MCS bin, between the FF and Transcoder.
• Figure 4: Proportion of aligned features as a function of the max-cosine score (MCS).
• Figure 5: Detailed feature alive scores on all tested models, across all layers.