Decomposing The Dark Matter of Sparse Autoencoders

TL;DR

The paper investigates the mysterious 'dark matter' of sparse autoencoders by showing that a large portion of SAE reconstruction error is linearly predictable from the input activations, and that per-token error scaling is also predictable across SAE scales. It decomposes SAE error into linear and nonlinear components, demonstrating that nonlinear error behaves qualitatively differently and is more difficult to learn, yet partly drives downstream cross-entropy increases. Through norm and vector prediction tests, component-wise SAE training, and interventions like inference-time gradient pursuit, the work outlines practical and theoretical implications for the Linear Representation Hypothesis and SAE design. The results suggest new directions for reducing nonlinear error and for circuit-level interpretability in language models.

Abstract

Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in "dark matter": unexplained variance in activations. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark matter -- about half of the error vector itself and >90% of its norm -- can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior of SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SAEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations. These insights imply that the part of the SAE error vector that cannot be linearly predicted ("nonlinear" error) might be fundamentally different from the linearly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worse, and 3) it is responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations are inserted into the model. Finally, we examine two methods to reduce nonlinear SAE error: inference time gradient pursuit, which leads to a very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to a larger reduction.

Figures (17)

• Figure 1: A breakdown of SAE dark matter for layer 20 Gemma 9B SAEs, with dotted lines assuming that observed trends continue for larger SAEs. See \ref{['sec:predicting_sae_error']} for how we break down the overall fraction of unexplained variance into absent features, linear error, and nonlinear error. See \ref{['sec:powerful-encoder']} for further separating encoder error from nonlinear error.
• Figure 1: Correlation matrix between synthetic noise and estimated errors.
• Figure 2: Results of linearly predicting SAE error norm and SAE error from model activations on Gemma 2 2B layer 12 (top) and Gemma 2 9B layer 20 (bottom). The right plots show the $R^2$ of predicting SAE error norms (see \ref{['eqn:probe_norm']}, the middle plots show the $R^2$ of predicting SAE error vectors (see \ref{['eqn:probe_vec']}, and the right plots show $1 - R^2$ of predicting model activations given the SAE reconstruction and the SAE error vector prediction. We note that these 2D heatmaps are somewhat sparse and only the black dots represent actual SAEs. This is because we use Gemma Scope SAEs, which are trained only on some $L_0$s and SAE widths. We do a linear interpolation between SAEs to predict $R^2$ between hyperparameters.
• Figure 3: $R^2$ of SAE error norm predictions (see \ref{['eqn:probe_norm']}) for Gemma Scope SAEs of width $L_0 \approx 50$ and Llama Scope SAEs of width $L_0 = k = 50$.
• Figure 4: $R^2$ for linear probes of per token SAE errors of larger SAEs from smaller SAEs. Prediction accuracy decreases as the SAEs get farther apart in scale, but overall remains high.