Forgetting is Competition: Rethinking Unlearning as Representation Interference in Diffusion Models

TL;DR

SurgUn (pronounced as Surgeon), a surgical unlearning method that applies targeted weight-space updates to remove specific visual concepts in text-conditioned diffusion models by inducing retroactive concept interference, enabling focused destabilization of only the target concept while preserving unrelated capabilities through a novel training paradigm.

Abstract

Unlearning in text-to-image diffusion models often leads to uneven concept removal and unintended forgetting of unrelated capabilities. This complicates tasks such as copyright compliance, protected data mitigation, artist opt-outs, and policy-driven content updates. As models grow larger and adopt more diverse architectures, achieving precise and selective unlearning while preserving generative quality becomes increasingly challenging. We introduce SurgUn (pronounced as Surgeon), a surgical unlearning method that applies targeted weight-space updates to remove specific visual concepts in text-conditioned diffusion models. Our approach is motivated by retroactive interference theory, which holds that newly acquired memories can overwrite, suppress, or impede access to prior ones by competing for shared representational pathways. We adapt this principle to diffusion models by inducing retroactive concept interference, enabling focused destabilization of only the target concept while preserving unrelated capabilities through a novel training paradigm. SurgUn achieves high-precision unlearning across diverse settings. It performs strongly on compact U-Net based models such as Stable Diffusion v1.5, scales effectively to the larger U-Net architecture SDXL, and extends to SANA, representing an underexplored Diffusion Transformer based architecture for unlearning.

Figures (41)

• Figure 1: Qualitative comparison of object‑unlearning methods evaluated on unlearning effectiveness (UE), in‑domain retainability (IDR), and cross‑domain retainability (CDR). In prompts, red text marks the unlearning target and green text marks retained concepts. Green‑framed images indicate correct behavior; red‑framed images indicate failures.
• Figure 2: Architecture of proposed method SurgUn, illustrated for U-Net–based diffusion models. We unlearn a target concept by selectively training chosen U-Net transformer blocks using our distractor-conditioned loss $\mathcal{L}_{\text{unlearn}}$, while all other blocks remain frozen. The example shown unlearns the concept "Bird."
• Figure 3: Distribution of distractor concepts across five categories, along with examples and their coverage in the distractor set.
• Figure 4: Hyperparameter search for weight-space localization, distractor concept selection, and checkpoint calibration using the SDXL model. Sub-figures (a), (b), and (c) show the UA scores and the average of IRA and CRA scores . In sub-figures (a) and (b), U-Net denotes training all blocks of the U-Net. The best-performing block identified in (a) is mid_block.0.attentions.0, and in (b) it is up_blocks.0.attentions.2. For weight-space localization, we observe a relatively higher variance in RA scores compared to UA. The variance for UA and RA in (a) is 66.87 and 82.90, respectively, and in (b) is 59.12 and 80.90, respectively. For the corresponding results for SD v1.5 and SANA, refer to Figure \ref{['fig: hyper_sd15']} and \ref{['fig: hyper_sana']} respectively in the Appendix.
• Figure 5: Sequential unlearning of multiple concepts. We present the average of unlearning accuracy (UA) and retaining accuracy (IRA and CRA). SurgUn achieves stronger unlearning while maintaining model performance, in contrast to competing approaches whose retention degrades. Quantitative results are presented in Table \ref{['tab: continual_appendix']} in the Appendix.