Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, Joseph Bloom

TL;DR

This work reveals a fundamental failure mode in sparse feature extraction for interpreting LLMs: feature absorption, where hierarchical relationships between features cause a single SAE latent to suffice for a signal while the intuitive, monosemantic latent fails to activate. The authors formalize absorption, provide a mathematical toy-model proof, and introduce a practical absorption metric to detect it across hundreds of SAEs and multiple models. They show that standard approaches like increasing width or tuning sparsity do not fully resolve absorption, and that feature splitting often accompanies absorption, complicating interpretability. The findings highlight limitations of current SAE-based interpretability and point to future directions such as Meta-SAEs and structured sparsity to robustly recover hierarchical feature structures in LLMs.

Abstract

Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features ("math" may split into "algebra", "geometry", etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get "absorbed" into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

TL;DR

This work reveals a fundamental failure mode in sparse feature extraction for interpreting LLMs: feature absorption, where hierarchical relationships between features cause a single SAE latent to suffice for a signal while the intuitive, monosemantic latent fails to activate. The authors formalize absorption, provide a mathematical toy-model proof, and introduce a practical absorption metric to detect it across hundreds of SAEs and multiple models. They show that standard approaches like increasing width or tuning sparsity do not fully resolve absorption, and that feature splitting often accompanies absorption, complicating interpretability. The findings highlight limitations of current SAE-based interpretability and point to future directions such as Meta-SAEs and structured sparsity to robustly recover hierarchical feature structures in LLMs.

Abstract

Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features ("math" may split into "algebra", "geometry", etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get "absorbed" into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.
Paper Structure (82 sections, 20 equations, 25 figures, 3 tables, 1 algorithm)

This paper contains 82 sections, 20 equations, 25 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Hierarchical features.
  4. Linear probing.
  5. K-sparse probing.
  6. Sparse autoencoders.
  7. SAE feature ablation.
  8. Toy models of feature absorption
  9. Independently firing features
  10. Hierarchical features cause absorption
  11. Proof: hierarchical features cause absorption
  12. Experimental setup
  13. Results
  14. Do SAEs learn latents that track first letter information?
  15. Why do SAE latents underperform?
  16. ...and 67 more sections

Figures (25)

  • Figure 1: In feature absorption, seemingly monosemantic latents fail to fire in cases where they apparently should. Here, we see an SAE can represent the word "short" and the concept "starts with S" more sparsely by absorbing the "starts with S" direction into the "short" latent, and then not firing the "starts with S" latent on the word "short", despite "short" starting with "S". Logical notation is used to describe the SAE encoder to emphasize its role as a classifier.
  • Figure 2: Comparison of independent features (left) vs. co-occurring features with absorption (right)
  • Figure 3: Interpretation of learned SAE latents with co-occurrence between feature 0 and feature 1 (feature 1 only fires if feature 0 fires).
  • Figure 4: Comparison of F1 scores for first-letter classification tasks
  • Figure 5: Precision and recall vs L0 for first-letter classification tasks
  • ...and 20 more figures