SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability

TL;DR

SAEBench tackles the challenge of evaluating sparse autoencoders for language-model interpretability by introducing an eight-metric benchmark that spans four core capabilities: Concept Detection, Interpretability, Reconstruction, and Feature Disentanglement. Through an open, extensible suite of over 200 SAEs and a standardized evaluation protocol, it demonstrates that gains on traditional sparsity–fidelity proxies do not reliably translate to practical interpretability, highlighting strong disentanglement performance for hierarchical Matryoshka architectures and complex scaling dynamics. The work provides actionable guidance for practitioners on sparsity settings, architecture choices, and training dynamics, and it lays the groundwork for broader, multi-modal and scale-aware SAE benchmarking. Overall, SAEBench advances mechanistic interpretability by offering a rigorous, reproducible lens to compare SAE architectures and training methods beyond single-metric optimization.

Abstract

Sparse autoencoders (SAEs) are a popular technique for interpreting language model activations, and there is extensive recent work on improving SAE effectiveness. However, most prior work evaluates progress using unsupervised proxy metrics with unclear practical relevance. We introduce SAEBench, a comprehensive evaluation suite that measures SAE performance across eight diverse metrics, spanning interpretability, feature disentanglement and practical applications like unlearning. To enable systematic comparison, we open-source a suite of over 200 SAEs across eight recently proposed SAE architectures and training algorithms. Our evaluation reveals that gains on proxy metrics do not reliably translate to better practical performance. For instance, while Matryoshka SAEs slightly underperform on existing proxy metrics, they substantially outperform other architectures on feature disentanglement metrics; moreover, this advantage grows with SAE scale. By providing a standardized framework for measuring progress in SAE development, SAEBench enables researchers to study scaling trends and make nuanced comparisons between different SAE architectures and training methodologies. Our interactive interface enables researchers to flexibly visualize relationships between metrics across hundreds of open-source SAEs at: www.neuronpedia.org/sae-bench

Figures (23)

• Figure 1: SAEBench evaluates sparse autoencoders across four fundamental capabilities. Concept Detection measures how well individual latents map to meaningful concepts. Interpretability evaluates feature comprehensibility using automated LLM evaluation. Reconstruction quantifies how faithfully the SAE preserves model behavior. Feature Disentanglement evaluates whether independent concepts are properly separated. These capabilities provide a comprehensive view of SAE performance beyond traditional metrics.
• Figure 2: Scores for the Loss Recovered, Automated Interpretability, Absorption, SCR, and Sparse Probing metrics on the 65k width Gemma-2-2B suite of SAEs.
• Figure 3: Scaling SAE width from 4k to 65k for across SAE architectures. For each architecture / width pair, we mean over all results in the $L_0$ range between 40 and 200. Most notably the hierarchical Matryoshka SAE shows positive scaling behavior. Due to varying L0 distributions across architectures, this visualization is intended primarily for analyzing scaling trends rather than architecture comparisons. Complete scaling results across all sparsity values are presented in Figure \ref{['fig:2x3_width_diff_scalingplot']}.
• Figure 4: Evaluation results for SAEs trained on the randomly initialized and final versions of Pythia-1B.
• Figure 5: Auto-interp scores for the canonical GemmaScope SAEs when compared to MLP neuron, PCA, and residual stream baselines. SAEs are significantly more interpretable than the baselines.