Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures

TL;DR

This work tests the Universality hypothesis by comparing Transformer- and Mamba-based language models using Sparse Autoencoders to extract interpretable, architecture-agnostic features. It introduces the Max Pairwise Pearson Correlation (MPPC) as a measure of cross-architecture feature similarity and demonstrates substantial feature-level universality, with SAE-derived features showing strong cross-arch correspondence and depth-wise alignment. At the circuit level, the authors reveal that induction mechanisms in Mamba closely mirror Transformer induction circuits, while also identifying an Off-by-One motif unique to Mamba. The study highlights both the promise and limitations of SAE-based universality: features form a large, common basis across architectures, yet some complex or uninterpretable patterns remain challenging to align, underscoring the need for deeper, automated circuit analyses. Overall, the results support a broad notion of architectural universality in language-model interpretability, with SAEs providing a practical, interpretable lens for cross-architecture comparison and circuit-level insights.

Abstract

The hypothesis of Universality in interpretability suggests that different neural networks may converge to implement similar algorithms on similar tasks. In this work, we investigate two mainstream architectures for language modeling, namely Transformers and Mambas, to explore the extent of their mechanistic similarity. We propose to use Sparse Autoencoders (SAEs) to isolate interpretable features from these models and show that most features are similar in these two models. We also validate the correlation between feature similarity and Universality. We then delve into the circuit-level analysis of Mamba models and find that the induction circuits in Mamba are structurally analogous to those in Transformers. We also identify a nuanced difference we call \emph{Off-by-One motif}: The information of one token is written into the SSM state in its next position. Whilst interaction between tokens in Transformers does not exhibit such trend.

Figures (13)

• Figure 1: Architecture of a MambaBlock. Some linear transformations and activation functions are omitted.
• Figure 2: hierarchy of invariance
• Figure 3: (a) Maximum correlation distribution between our main experiment, neuronal approach, seed variant skyline and identical model skyline. (b) Distribution of the difference in MPPC calculated by Mamba and Pythia (model seed variant). (c) Frequency of the best matching feature pairs falling on each layer. Matched pairs mainly fall on near depth, i.e. on the diagonal.
• Figure 4: We present four cases of feature pairs with different Pearson correlation values. To the right of each feature index is the auto-interpretation result generated by a large language model (LLM), with activation examples shown below.
• Figure 5: (a) Distribution of MPPC over auto-labeled complexity scores ranging from 1 (simple) to 4 (complex). Both Model Seed Variant and Cross-Arch SAE MPPC exhibit correlation, while one in SAE Seed Variant is weaker. (b) Distribution of MPPC over monosemanticity scores ranging from 1 (No) to 2 (Yes). Monosemanticity scores almost divide MPPC into two interval categories at about 0.2.