Understanding Empirical Unlearning with Combinatorial Interpretability

TL;DR

This paper investigates empirical unlearning through combinatorial interpretability on two-layer networks, reproducing several erasure methods and evaluating whether erasure truly eliminates a target concept or only suppresses it. It shows that erased knowledge persists in weights and can resurface during fine-tuning, even with unrelated data, and that recovery often follows the original encoding direction rather than random drift. By decomposing recovery updates, the work reveals a dominant parallel component to the original erasure direction, suggesting directed reversal rather than stochastic drift. These findings provide mechanistic insight into why erased knowledge can resurface and offer a principled framework for evaluating unlearning methods on interpretable, simplified models with implications for larger foundation models.

Abstract

While many recent methods aim to unlearn or remove knowledge from pretrained models, seemingly erased knowledge often persists and can be recovered in various ways. Because large foundation models are far from interpretable, understanding whether and how such knowledge persists remains a significant challenge. To address this, we turn to the recently developed framework of combinatorial interpretability. This framework, designed for two-layer neural networks, enables direct inspection of the knowledge encoded in the model weights. We reproduce baseline unlearning methods within the combinatorial interpretability setting and examine their behavior along two dimensions: (i) whether they truly remove knowledge of a target concept (the concept we wish to remove) or merely inhibit its expression while retaining the underlying information, and (ii) how easily the supposedly erased knowledge can be recovered through various fine-tuning operations. Our results shed light within a fully interpretable setting on how knowledge can persist despite unlearning and when it might resurface.

Figures (2)

• Figure 1: Weights of a 2-layer neural network trained on the Boolean formula in Eq.\ref{['eq:dnf-example']}. Row numbers indicate neurons, and column numbers in Layer 1 indicate input variables. "Positive" neurons (positive values in $W_2$) encode different clauses, with their corresponding weights being markedly positive. "Negative" neurons suppress spurious activations to inhibit misclassification.
• Figure 2: Model weights of original trained model and models after concept erasure. Models erased using Task Vectors or Gradient Ascent remain very close to the original model, whereas PPD yields a model with substantially different internal encodings because it is trained as a student network using random initialization.