FADE: Selective Forgetting via Sparse LoRA and Self-Distillation

TL;DR

This work tackles selective unlearning in text-to-image diffusion models under regulatory and safety constraints. FADE introduces a two-stage approach: first, a knowledge-location step uses gradient-based saliency to confine updates to a sparse set of parameters via LoRA adapters; second, a self-distillation stage overwrites the forgotten concept with a user-defined surrogate, guided by a specialized loss and conditional prompts. The adapters are memory-efficient, mergeable at inference time, and enable reversible deployment, yielding strong forgetting with high retainability on benchmarks like UnlearnCanvas, supported by ablations on multiple datasets. Overall, FADE offers a practical, controllable, and scalable solution for selective unlearning in diffusion-based image generation with broad production applicability.

Abstract

Machine Unlearning aims to remove the influence of specific data or concepts from trained models while preserving overall performance, a capability increasingly required by data protection regulations and responsible AI practices. Despite recent progress, unlearning in text-to-image diffusion models remains challenging due to high computational costs and the difficulty of balancing effective forgetting with retention of unrelated concepts. We introduce FADE (Fast Adapter for Data Erasure), a two-stage unlearning method for image generation that combines parameter localization with self-distillation. FADE first identifies parameters most responsible for the forget set using gradient-based saliency and constrains updates through sparse LoRA adapters, ensuring lightweight, localized modifications. In a second stage, FADE applies a self-distillation objective that overwrites the forgotten concept with a user-defined surrogate while preserving behavior on retained data. The resulting adapters are memory-efficient, reversible, and can be merged or removed at runtime, enabling flexible deployment in production systems. We evaluated FADE on the UnlearnCanvas benchmark and conducted ablation studies on Imagenette, Labeled Faces in the Wild, AtharvaTaras Dog Breeds Dataset, and SUN Attributes datasets, demonstrating State-of-the-Art unlearning performance with fine-grained control over the forgetting-retention trade-off. Our results demonstrate that FADE achieves strong concept erasure and high retainability across various domains, making it a suitable solution for selective unlearning in diffusion-based image generation models.

Figures (15)

• Figure 1: Schematic summary of our method. As shown on the left, binary mask will be computed with gradient based saliency and it will be used to make the product of the LoRA matrices sparse accordingly. The fine-tuning is done with self-distillation by toggling the LoRAs on and off as shown at the bottom. After fine-tuning, the LoRAs will be merged to the model for inference
• Figure 2: Unlearning accuracies by retaining accuracies for the methods shown in Table \ref{['tab:performance']}
• Figure 3: Peak memory during training by training runtime for the methods shown in Table \ref{['tab:performance']}
• Figure 4: Examples of object forgetting under the Gorgeous Love style. The left column describes the unlearned model concept, in between brackets the overwriting concept used, and in the first row the object used in the prompt to sample the image.
• Figure 5: Examples of styles forgetting evaluated with the trees class. The left column describes the unlearned model concept, in between brackets the overwriting concept used, and in the first row the style used in the prompt to sample the image.