Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features

TL;DR

This paper addresses the gap in SAE interpretability by introducing a weight-based, out-of-context framework that analyzes direct weight interactions to reveal what SAE features do, independent of activation data. It formalizes features via encoder–decoder pairs and per-feature logit effects $l_D^i$, and identifies semantic features that causally predict tokens using three complementary metrics, calibrated at a representative depth. Across Gemma-2 and Llama-3.1 SAEs (100 instances), roughly $1/4$ of features are semantically coherent, exhibiting a depth-dependent distribution: a U-shaped pattern in tied embeddings and an inverted-U pattern for features specialized in attention, with additional input-encoding nuances in untied architectures. The study further demonstrates that many features participate in attention circuits, revealing a computational architecture beyond simple activation patterns and showing that semantic and non-semantic roles interact differently across layers and architectures, enabling more robust, causally grounded interpretability.

Abstract

Sparse autoencoders (SAEs) have emerged as a powerful technique for decomposing language model representations into interpretable features. Current interpretation methods infer feature semantics from activation patterns, but overlook that features are trained to reconstruct activations that serve computational roles in the forward pass. We introduce a novel weight-based interpretation framework that measures functional effects through direct weight interactions, requiring no activation data. Through three experiments on Gemma-2 and Llama-3.1 models, we demonstrate that (1) 1/4 of features directly predict output tokens, (2) features actively participate in attention mechanisms with depth-dependent structure, and (3) semantic and non-semantic feature populations exhibit distinct distribution profiles in attention circuits. Our analysis provides the missing out-of-context half of SAE feature interpretability.

Figures (34)

• Figure 1: An SAE feature with activated tokens highlighted and the highest activation values boxed. The activation-based generated by GPT-4o-mini is superficial and fails to capture the causal effect.
• Figure 2: A uniform sample of features that met all 3 thresholds, along with their top 10 tokens. "L" denotes Levenshtein similarity, "C" denotes cosine similarity, and "E" denotes top-100 entropy. The boxed scores correspond to the range of displayed tokens.
• Figure 3: Main results for Experiment 1 on Gemma-2-2B and 9B: Semantic features display a U-shaped distribution, with an average joint pass rate of 24.01% and 25.62%, respectively. Ablation using subsets of the 3 metrics leads to a stepwise decrease in pass rates across layers, demonstrating their complementary effectiveness.
• Figure 4: Main results for Experiment 1 on Llama-3.1-8B: Semantic features display monotonic distributions, with an average joint pass rate of 23.32% and 11.82%, respectively. Thresholds are obtained from layer 20 for the decoder--unembedding pair and from layer 5 for the encoder--embedding pair.
• Figure 5: Main results for Experiment 2 on Gemma-2-9B and Llama-3.1-8B: Magnitude of attention participation across layers, with the total height at each layer representing the aggregate pre-softmax score. These scores are calculated as the mean across all keys for each query feature, then summed for each head. The stacking direction of the head values is determined based on whether they are positive or negative. Each color represents a rank of magnitude, rather than a specific head index, across all layers.