Mass Concept Erasure in Diffusion Models with Concept Hierarchy

TL;DR

This work addresses the challenge of erasing multiple concepts in diffusion-based T2I models without sacrificing general generation. It introduces a two-level concept hierarchy and group-wise suppression to jointly erase semantically related leaves under shared supertypes, coupled with diffusion regularization to preserve denoising. The SuPLE?RA mechanism freezes the down-projection $B_j$ and trains only the up-projection $A_j$ to preserve supertype generation, with subspaces derived from supertype descriptions via SVD, and it fuses multiple modules through distillation to a final $W^*$. Across a cross-domain benchmark (celebrities, objects, porn content), the method achieves better erasure efficiency and generation preservation than prior approaches, offering improved scalability for mass concept erasure in diffusion models.

Abstract

The success of diffusion models has raised concerns about the generation of unsafe or harmful content, prompting concept erasure approaches that fine-tune modules to suppress specific concepts while preserving general generative capabilities. However, as the number of erased concepts grows, these methods often become inefficient and ineffective, since each concept requires a separate set of fine-tuned parameters and may degrade the overall generation quality. In this work, we propose a supertype-subtype concept hierarchy that organizes erased concepts into a parent-child structure. Each erased concept is treated as a child node, and semantically related concepts (e.g., macaw, and bald eagle) are grouped under a shared parent node, referred to as a supertype concept (e.g., bird). Rather than erasing concepts individually, we introduce an effective and efficient group-wise suppression method, where semantically similar concepts are grouped and erased jointly by sharing a single set of learnable parameters. During the erasure phase, standard diffusion regularization is applied to preserve denoising process in unmasked regions. To mitigate the degradation of supertype generation caused by excessive erasure of semantically related subtypes, we propose a novel method called Supertype-Preserving Low-Rank Adaptation (SuPLoRA), which encodes the supertype concept information in the frozen down-projection matrix and updates only the up-projection matrix during erasure. Theoretical analysis demonstrates the effectiveness of SuPLoRA in mitigating generation performance degradation. We construct a more challenging benchmark that requires simultaneous erasure of concepts across diverse domains, including celebrities, objects, and pornographic content.

Figures (8)

• Figure 1: As more celebrities are erased, Stable Diffusion shows clear degradation in generating the supertype person. Results are shown for three representative celebrities (“Adam Driver”, “Adriana Lima”, “Amber Heard”). Green boxes in the first row indicate successful erasure. In the second row, green boxes indicate the generation capability of the supertype is preserved, while red boxes indicate failure.
• Figure 2: An overview of our mass concept erasure framework. We first construct a supertype–subtype concept hierarchy using LLMs (Sec.\ref{['sec:hierarchy']}). Leveraging this hierarchy, we introduce a group-wise suppression method that jointly erases grouped concepts via MACE’s attention modulation, and apply diffusion regularization to preserve the model’s denoising capability (Sec.\ref{['sec:suppression']}). To mitigate the degradation of supertype generation caused by excessive erasure of semantically related subtypes, we propose SuPLoRA, a supertype-preserving low-rank adaptation that freezes the down-projection matrix $\bm{B}_j$ (orthogonal to the supertype subspace) and updates only the up-projection matrix $\bm{A}_j$, as theoretically analyzed in Sec.\ref{['sec:SuPLoRA']}. Finally, we merge multiple SuPLoRA modules into a unified representation while preserving general generation ability (Sec. \ref{['sec:SuPLoRA']}).
• Figure 3: Qualitative comparison of mass concept erasure. Green boxes indicate successful removal of erased concepts and successful retention of preserved concepts. Red boxes highlight undesirable cases.
• Figure 4: Qualitative comparison of erasing targeted concepts in the celebrity domain.
• Figure 5: Qualitative comparison of erasing targeted concepts in the object domain.