Toward Understanding Unlearning Difficulty: A Mechanistic Perspective and Circuit-Guided Difficulty Metric

TL;DR

This work tackles why unlearning difficulty varies across samples by proposing a circuit-based, pre-unlearning score called Circuit-guided Unlearning Difficulty (CUD). CUD derives from two reference circuits representing easy and hard forgetting, enabling offline stratification of samples and benchmarking across unlearning methods. Empirically, easy-forget samples rely on shallow, early-to-intermediate circuit edges, while hard-forget samples depend on deeper, late-stage and attention-related pathways, with robust performance across similarity metrics and loss functions. The results support a mechanistic interpretation of forgetting, enabling difficulty-aware unlearning, targeted interventions, and principled benchmarking. Overall, CUD offers a principled, interpretable lens on how model internals govern memorized knowledge and its erasure.

Abstract

Machine unlearning is becoming essential for building trustworthy and compliant language models. Yet unlearning success varies considerably across individual samples: some are reliably erased, while others persist despite the same procedure. We argue that this disparity is not only a data-side phenomenon, but also reflects model-internal mechanisms that encode and protect memorized information. We study this problem from a mechanistic perspective based on model circuits--structured interaction pathways that govern how predictions are formed. We propose Circuit-guided Unlearning Difficulty (CUD), a {\em pre-unlearning} metric that assigns each sample a continuous difficulty score using circuit-level signals. Extensive experiments demonstrate that CUD reliably separates intrinsically easy and hard samples, and remains stable across unlearning methods. We identify key circuit-level patterns that reveal a mechanistic signature of difficulty: easy-to-unlearn samples are associated with shorter, shallower interactions concentrated in earlier-to-intermediate parts of the original model, whereas hard samples rely on longer and deeper pathways closer to late-stage computation. Compared to existing qualitative studies, CUD takes a first step toward a principled, fine-grained, and interpretable analysis of unlearning difficulty; and motivates the development of unlearning methods grounded in model mechanisms.

Figures (5)

• Figure 1: A mechanistic perspective of unlearning difficulty. (a) Illustration of unlearning and post-unlearning performance differences. (b) An example of mechanistic differences of circuits between easy- and hard-to-unlearn samples. (c) Proposed Circuit-guided Unlearning Difficulty (CUD) score to quantify unlearning difficulty of samples.
• Figure 2: CUD score distribution of samples from TOFU. Red: Default forget set. Blue: All samples. The default forget samples closely matches the overall distribution, with a wide coverage of all difficulty levels, suggesting that TOFU's default forget set represents a mid-level unlearning difficulty. Using CUD, we can select a harder / easier forget sets than the default to control task difficulty and better stress-test unlearning methods.
• Figure 3: CUD score is robust to choices of $L_{\text{MU}}$ in Eq. \ref{['eq:find_easy']} and \ref{['eq:find_hard']}, since the regularizer leads to sparse and consistent selection of representative samples. Correlation coefficient $\rho=0.76$. Each point represent in the figure represent the CUD scores computed using the GradDiff maini2024tofu MU loss and the UNDIAL dong2024undial MU loss.
• Figure 4: Edge distribution of statistically different easy and hard circuits.
• Figure 5: Comparison between CUD-based and MRD-based difficulty score on TOFU. CUD and MRD captures fundamentally different information when quantifying sample unlearning difficulty, with a correlation coefficient $\rho=-0.27$.