Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

Maheep Chaudhary, Atticus Geiger

TL;DR

This work uses the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in and shows that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline.

Abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

TL;DR

This work uses the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in and shows that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline.

Abstract

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel
Paper Structure (27 sections, 7 equations, 3 figures, 2 tables)

This paper contains 27 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarking SAEs
  4. Interpretability of Knowledge Representations
  5. Methodology
  6. The RAVEL Benchmark
  7. Filtering
  8. Interchange Interventions in Feature Space
  9. Counterfactual Labels
  10. Splits
  11. Constructing and Selecting Features
  12. Sparse Autoencoders for Dictionary Learning
  13. Distributed Alignment Search
  14. Differential Binary Masking
  15. Experiments
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Metrics on the RAVEL test set for interchange interventions performed on the residual stream in GPT-2 small after transformer block above the city token <city>. For each space of features, we learn 'country' features that encode what country a city is in and 'continent' features that encode what continent a city is in. Interventions targeting the 'country' features should change the output for the prompt <city> is in the country of, but not <city> is in the continent of. Interventions targeting the 'continent' features should do the opposite. The disentangle score is the average of the country and continent accuracies. Neurons serve as a baseline for how easily these two facts are disentangled, and DAS is a supervised feature learning method that serves as a skyline. The SAEs are the methods we seek to evaluate. In sum, using SAE reconstructions harm the knowledge of GPT-2 and SAE features are not better mediators than the baseline of neurons.
  • Figure :
  • Figure :