CRISP: Persistent Concept Unlearning via Sparse Autoencoders

TL;DR

CRISP introduces a persistent, parameter-efficient unlearning method for LLMs by automatically identifying and suppressing salient SAE-derived features tied to a target concept. It combines contrastive target/retain activation analysis with LoRA-based fine-tuning, optimizing for unlearning, retention, and coherency to achieve state-of-the-art performance on WMDP safety benchmarks. Feature-level analyses demonstrate semantically coherent activation directions aligned with the target concept, supporting interpretability and targeted interventions. Across two open-weight models and two domains (biosecurity and cybersecurity), CRISP delivers strong forgetting while preserving benign knowledge and generation quality, suggesting practical utility for safe model deployment and release. The work also outlines limitations related to SAE dependence and formal guarantees, charting future work on generalization and theoretical bounds.

Abstract

As large language models (LLMs) are increasingly deployed in real-world applications, the need to selectively remove unwanted knowledge while preserving model utility has become paramount. Recent work has explored sparse autoencoders (SAEs) to perform precise interventions on monosemantic features. However, most SAE-based methods operate at inference time, which does not create persistent changes in the model's parameters. Such interventions can be bypassed or reversed by malicious actors with parameter access. We introduce CRISP, a parameter-efficient method for persistent concept unlearning using SAEs. CRISP automatically identifies salient SAE features across multiple layers and suppresses their activations. We experiment with two LLMs and show that our method outperforms prior approaches on safety-critical unlearning tasks from the WMDP benchmark, successfully removing harmful knowledge while preserving general and in-domain capabilities. Feature-level analysis reveals that CRISP achieves semantically coherent separation between target and benign concepts, allowing precise suppression of the target features.

Figures (5)

• Figure 1: Overview of CRISP: (1) We identify features that are frequently and strongly activated by the target corpus---but not by the benign corpus---using pre-trained sparse autoencoders (SAEs). (2) We then fine-tune the model to suppress these features on the target corpus, while preserving their activations on the benign corpus.
• Figure 2: Qualitative comparison of generations after different unlearning methods. We prompt about non-harmful biomedical knowledge that is topically related to harmful concepts from the WMDP-Bio dataset. While existing methods disrupt fluency or inject artifacts (e.g., repetition, formatting tokens), CRISP retains coherent and informative generations, demonstrating effective preservation of general-domain capabilities.
• Figure 3: Trade-off between Retain accuracy (y-axis) and Unlearn accuracy (x-axis) on the WMDP-Bio benchmark. Each point represents one of $200$ hyperparameter configurations per method. The red star marks the ideal point: random guessing on the unlearning benchmark with unchanged retain accuracy. The solid envelope line connects the best configuration in each unlearning accuracy bucket, illustrating the Pareto frontier.
• Figure 4: Feature distributions across benign (x-axis) and target (y-axis) activation frequencies. Each point represents a feature, with color intensity indicating the target-to-benign activation ratio. Points along the diagonal have similar activation rates for both datasets (circled in purple). Salient target features (circled in red) appear in the upper-left region, while salient benign features (circled in blue) appear in the lower-right.
• Figure 5: Trade-off between Retain Accuracy (y-axis) and Unlearn Accuracy (x-axis) on the WMDP-Cyber benchmark. Top: Llama-3.1-8B, Bottom: Gemma-2-2B. Each point shows one of 200 hyperparameter settings per method. The red star indicates the ideal outcome—complete forgetting with no loss in retain accuracy. The solid line traces the best result per unlearning bucket, forming the Pareto frontier.