Sparse Autoencoders Can Interpret Randomly Initialized Transformers

TL;DR

This paper applies SAEs to 'interpret' random transformers, i.e., transformers where the parameters are sampled IID from a Gaussian rather than trained on text data, and finds that random and trained transformers produce similarly interpretable SAE latents.

Abstract

Sparse autoencoders (SAEs) are an increasingly popular technique for interpreting the internal representations of transformers. In this paper, we apply SAEs to 'interpret' random transformers, i.e., transformers where the parameters are sampled IID from a Gaussian rather than trained on text data. We find that random and trained transformers produce similarly interpretable SAE latents, and we confirm this finding quantitatively using an open-source auto-interpretability pipeline. Further, we find that SAE quality metrics are broadly similar for random and trained transformers. We find that these results hold across model sizes and layers. We discuss a number of number interesting questions that this work raises for the use of SAEs and auto-interpretability in the context of mechanistic interpretability.

Figures (20)

• Figure 1: ROC curves for 'fuzzing' auto-interpretability for Pythia-410M over 100 SAE latents. These results demonstrate the similarity in performance between the SAE variants, as well as the overall degradation in performance as the layer index increases. The auto-interpretability scores here fail to distinguish between trained and randomized models.
• Figure 2: ROC curves for 'fuzzing' auto-interpretability for Pythia-6.9B over 100 SAE latents. Similarly to Figure \ref{['fig:pythia_410m_fuzzing']}, these results demonstrate the similarity in performance between the SAE variants variants. Notably, the trained variant appears to degrade in performance for the later model layers.
• Figure 3: Comparison of sparse autoencoder performance across Pythia models (70M to 6.9B parameters). The different SAE variants show remarkably similar trends across model scales, with larger models exhibiting more consistent behavior across layers. All variants save for control achieve comparable performance despite fundamentally different initialization approaches.
• Figure 4: An illustrative example of the effect of a randomly initialized neural network on superposed input data. We take 10K samples of ${n_s}=3$ sparse input features from a Lomax distribution with shape $\alpha=1$ and scale $\lambda=1$ and project these to ${n_d}=2$ dense input features by an IID standard normal matrix. Then, we pass the dense outputs to a two-layer MLP with ReLU activation and hidden size of $4{n_d}$ and recover ${n_s}=3$ sparse outputs by the inverse of the previously generated projection matrix.
• Figure 5: The mean max cosine similarity (MMCS) between the features learned by a standard SAE (decoder weight vectors) and the data-generating features against the $L^1$ penalty coefficient in the training loss, following sharkey_taking_2022. There is a 'Goldilocks zone' where SAEs near-perfectly recover the data-generating features, given enough latents to represent them.