Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Applying sparse autoencoders to unlearn knowledge in language models

Eoin Farrell, Yeu-Tong Lau, Arthur Conmy

TL;DR

This work investigates using sparse autoencoders (SAEs) to perform interpretable unlearning of hazardous knowledge in language models. By training SAEs on OpenWebText and intervening through negative clamping of selected residual-stream features, the study evaluates unlearning on the WMDP-bio subset using two Gemma models, comparing against RMU fine-tuning. Results show that individual bio-related SAE features can reduce WMDP-bio accuracy with minimal collateral damage, but zero ablation is ineffective and multi-feature interventions introduce comparable or larger side-effects than RMU; overall, SAE-based unlearning currently underperforms fine-tuning-based approaches. The findings highlight the potential and limitations of SAE-based unlearning and point to future work on SAE quality, feature width, and cross-layer dynamics to achieve stronger, more reliable knowledge removal. The work contributes toward more transparent, verifiable mechanisms for content-filtering in language models.

Abstract

We investigate whether sparse autoencoders (SAEs) can be used to remove knowledge from language models. We use the biology subset of the Weapons of Mass Destruction Proxy dataset and test on the gemma-2b-it and gemma-2-2b-it language models. We demonstrate that individual interpretable biology-related SAE features can be used to unlearn a subset of WMDP-Bio questions with minimal side-effects in domains other than biology. Our results suggest that negative scaling of feature activations is necessary and that zero ablating features is ineffective. We find that intervening using multiple SAE features simultaneously can unlearn multiple different topics, but with similar or larger unwanted side-effects than the existing Representation Misdirection for Unlearning technique. Current SAE quality or intervention techniques would need to improve to make SAE-based unlearning comparable to the existing fine-tuning based techniques.

Applying sparse autoencoders to unlearn knowledge in language models

TL;DR

This work investigates using sparse autoencoders (SAEs) to perform interpretable unlearning of hazardous knowledge in language models. By training SAEs on OpenWebText and intervening through negative clamping of selected residual-stream features, the study evaluates unlearning on the WMDP-bio subset using two Gemma models, comparing against RMU fine-tuning. Results show that individual bio-related SAE features can reduce WMDP-bio accuracy with minimal collateral damage, but zero ablation is ineffective and multi-feature interventions introduce comparable or larger side-effects than RMU; overall, SAE-based unlearning currently underperforms fine-tuning-based approaches. The findings highlight the potential and limitations of SAE-based unlearning and point to future work on SAE quality, feature width, and cross-layer dynamics to achieve stronger, more reliable knowledge removal. The work contributes toward more transparent, verifiable mechanisms for content-filtering in language models.

Abstract

We investigate whether sparse autoencoders (SAEs) can be used to remove knowledge from language models. We use the biology subset of the Weapons of Mass Destruction Proxy dataset and test on the gemma-2b-it and gemma-2-2b-it language models. We demonstrate that individual interpretable biology-related SAE features can be used to unlearn a subset of WMDP-Bio questions with minimal side-effects in domains other than biology. Our results suggest that negative scaling of feature activations is necessary and that zero ablating features is ineffective. We find that intervening using multiple SAE features simultaneously can unlearn multiple different topics, but with similar or larger unwanted side-effects than the existing Representation Misdirection for Unlearning technique. Current SAE quality or intervention techniques would need to improve to make SAE-based unlearning comparable to the existing fine-tuning based techniques.

Paper Structure

This paper contains 20 sections, 10 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Models and Dataset
  4. Unlearning metrics
  5. Intervention Methods
  6. How to select relevant SAE Features
  7. Unlearning using a single feature
  8. A Representative Feature (#9163)
  9. Impact of clamping feature activation
  10. Permutations
  11. Evaluation of SAE Unlearning
  12. Results
  13. Impact of encoder vs decoder in unlearning
  14. Conclusion
  15. Multiple Choice Question Considerations
  16. ...and 5 more sections

Figures (10)

  • Figure 1: An outline of how we use SAE features to intervene in the model. Selected feature activations $f_{i}$ are set to a negative value $-c$ when $f_i > 0$.
  • Figure 2: Max-activating prompts from OpenWebText (blue) and the WMDP-bio dataset (green) for a selected representative SAE feature #9163 in https://huggingface.co/google/gemma-2b-it at layer 9. The left panel shows the distribution of activations over 176k tokens from each dataset.
  • Figure 3: Content of question #841 from the WMDP-bio dataset with highlights indicating the strength of the activation of feature #9163 on each token in the prompt
  • Figure 4: Probabilities of answering A, B, C or D for question #841 (with correct answer A) as a function of the clamped activation value of feature #9163. Loss added is calculated over 50k tokens of OpenWebText. Question #841 is presented in \ref{['question_841']}.
  • Figure 5: Unlearning performance comparison for SAEs with different numbers of intervened features (10, 20, 50) and RMU on https://huggingface.co/google/gemma-2-2b-it. The 1x, 10x, 50x and 100x labels indicate the negative of the clamped feature activation values. Top: Loss added (+ 0.005 for clarity) vs. WMDP-bio Accuracy. Bottom: Selected MMLU Accuracy vs. WMDP-bio Accuracy. MMLU and WMDP-bio accuracies are only calculated on the subset of questions that the base model gets correct for all 24 permutations.
  • ...and 5 more figures