Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking

TL;DR

This paper tackles the challenge of multi-concept unlearning in text-to-image diffusion models by introducing Forget-It-All (FIA), a training-free, sparsity-driven framework. FIA combines Contrastive Concept Saliency, time- and space-based Concept-Sensitive Neuron identification, and a Multi-concept Mask Fusion strategy that preserves Concept-Agnostic Neurons to maintain image quality while pruning concept-specific components. Across object removal, explicit content suppression, and artist-style forgetting, FIA achieves state-of-the-art forgetting with minimal pruning (<0.3%) and strong semantic fidelity, demonstrating robustness and practical applicability for copyright and safety considerations. The work offers a plug-and-play solution with minimal hyperparameter tuning, scalable to real-world unlearning needs while highlighting areas for improvement at very large concept counts.

Abstract

The widespread adoption of text-to-image (T2I) diffusion models has raised concerns about their potential to generate copyrighted, inappropriate, or sensitive imagery learned from massive training corpora. As a practical solution, machine unlearning aims to selectively erase unwanted concepts from a pre-trained model without retraining from scratch. While most existing methods are effective for single-concept unlearning, they often struggle in real-world scenarios that require removing multiple concepts, since extending them to this setting is both non-trivial and problematic, causing significant challenges in unlearning effectiveness, generation quality, and sensitivity to hyperparameters and datasets. In this paper, we take a unique perspective on multi-concept unlearning by leveraging model sparsity and propose the Forget It All (FIA) framework. FIA first introduces Contrastive Concept Saliency to quantify each weight connection's contribution to a target concept. It then identifies Concept-Sensitive Neurons by combining temporal and spatial information, ensuring that only neurons consistently responsive to the target concept are selected. Finally, FIA constructs masks from the identified neurons and fuses them into a unified multi-concept mask, where Concept-Agnostic Neurons that broadly support general content generation are preserved while concept-specific neurons are pruned to remove the targets. FIA is training-free and requires only minimal hyperparameter tuning for new tasks, thereby promoting a plug-and-play paradigm. Extensive experiments across three distinct unlearning tasks demonstrate that FIA achieves more reliable multi-concept unlearning, improving forgetting effectiveness while maintaining semantic fidelity and image quality.

Figures (14)

• Figure 1: The proposed FIA framework enables simultaneous multi-concept unlearning in text-to-image models. In this figure, we demonstrate the unlearning effects of 2 concepts in the multi-concept unlearning scenario with FIA (more comprehensive results are shown in Section \ref{['exp']}). It shows that FIA can (i) unlearn multiple undesired objects, (ii) prevent the generation of explicit content, and (iii) mitigate artwork copyright issues. This figure illustrates that FIA not only achieves robust multi-concept unlearning but also preserves the generation quality.
• Figure 2: Overview of our unlearning framework (illustrated with golf ball, French horn, and church). We first compute Contrastive Concept Saliency to quantify neuron responses to target concepts. These scores are aggregated over time and refined with spatial sparsity to identify Concept-Sensitive Neurons. Finally, we generate per-concept masks and fuse them into a multi-concept mask while preserving concept-agnostic neurons.
• Figure 3: Visual results on the Imagenette dataset, demonstrating simultaneous unlearning of five target classes while preserving the other five. Our method achieves superior unlearning performance on the target classes, and continues to faithfully generate the preserved classes. More visual results can be found in Figure \ref{['app:obj1']},\ref{['app:obj2']}.
• Figure 4: Forgetting accuracy (a) and Overall Score (b) on Imagenette across various forget–preserve configurations, demonstrating FIA’s superior balance between unlearning efficacy and generation quality.
• Figure 4: Comparison of unlearning methods for simultaneous unlearning of five artist styles.