Forget Many, Forget Right: Scalable and Precise Concept Unlearning in Diffusion Models

TL;DR

This paper tackles large-scale unlearning in diffusion models by introducing ScaPre, a training-free framework that unlearns many concepts while preserving broad image quality. It combines a conflict-aware stable design with a geometry-preserving proximal refinement and an Informax Decoupler to confine updates to concept-relevant subspaces, solved in closed form via a Sylvester equation and a Bures-distance–guided refinement. Empirical results show ScaPre can forget up to about five times more concepts than strong baselines (with comparable generation quality) across objects, styles, and explicit-content prompts, demonstrating strong precision and scalability. The approach offers a practical, efficient path toward safe and controllable diffusion models suitable for large-scale concept removal in real-world deployments.

Abstract

Text-to-image diffusion models have achieved remarkable progress, yet their use raises copyright and misuse concerns, prompting research into machine unlearning. However, extending multi-concept unlearning to large-scale scenarios remains difficult due to three challenges: (i) conflicting weight updates that hinder unlearning or degrade generation; (ii) imprecise mechanisms that cause collateral damage to similar content; and (iii) reliance on additional data or modules, creating scalability bottlenecks. To address these, we propose Scalable-Precise Concept Unlearning (ScaPre), a unified framework tailored for large-scale unlearning. ScaPre introduces a conflict-aware stable design, integrating spectral trace regularization and geometry alignment to stabilize optimization, suppress conflicts, and preserve global structure. Furthermore, an Informax Decoupler identifies concept-relevant parameters and adaptively reweights updates, strictly confining unlearning to the target subspace. ScaPre yields an efficient closed-form solution without requiring auxiliary data or sub-models. Comprehensive experiments on objects, styles, and explicit content demonstrate that ScaPre effectively removes target concepts while maintaining generation quality. It forgets up to $\times \mathbf{5}$ more concepts than the best baseline within acceptable quality limits, achieving state-of-the-art precision and efficiency for large-scale unlearning.

Figures (13)

• Figure 1: As the total number of unlearned concepts increases, existing methods either fail to effectively forget the target concepts or suffer from severe and noticeable degradation in overall image generation quality. In contrast, our proposed method ScaPre consistently maintains both stable unlearning performance and high generative quality (left). Moreover, ScaPre does not compromise similar non-target concepts, thereby clearly demonstrating its precise unlearning capability (right).
• Figure 2: Overview of ScaPre. Given target concepts $C_E$, cross-attention representations are first regularized by the spectral trace regularizer $\mathcal{L}_t$, while the informax decoupler $\bm{\alpha}$ confines updates to concept-relevant subspaces. Within this stabilized space, closed-form optimization yields $W^\star$, realizing the forgetting of $C_E$ (red arrows). Geometry alignment $\mathcal{L}_g$ then applies a proximal refinement toward the pretrained reference $W_0$, preserving global structure (green arrows). This staged pipeline enables scalable, precise unlearning while maintaining non-target generation quality.
• Figure 2: We report evaluation across multiple metrics including CLIP$_{art}$, CLIP$_{coco}$, CLIP$_x$ (difference), and FID. ScaPre consistently outperforms baselines.
• Figure 3: We demonstrate the overall results on the ImageNet-Diversi50, including Avg Acc, CLIP Score and UQ. ScaPre achieves significantly better performance compared to all baselines.
• Figure 4: We compare how unlearning accuracy ($\downarrow$) and UQ ($\uparrow$) change as the number of concepts increases across methods. ScaPre consistently achieves the best performance.