CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabulary

TL;DR

CE-SDWV tackles the problem of erasing undesired concepts from text-to-image diffusion models without retraining. It introduces a semantic-driven vocabulary approach that builds a $k$-dimensional target-concept subspace from a token matrix and then applies adaptive suppression at the token level, complemented by a gradient-orthogonal optimization to align edits with the original image space. The method demonstrates strong erasure performance across sexual, object, and style concepts on I2P and UnlearnCanvas, with robustness to adversarial prompts and applicability across SD model variants, while maintaining high image quality and efficiency. These contributions offer a practical, scalable path toward safer diffusion-based generation, though the authors acknowledge residual inconsistencies and note the potential for extending to multi-concept erasure in future work.

Abstract

Large-scale text-to-image (T2I) diffusion models have achieved remarkable generative performance about various concepts. With the limitation of privacy and safety in practice, the generative capability concerning NSFW (Not Safe For Work) concepts is undesirable, e.g., producing sexually explicit photos, and licensed images. The concept erasure task for T2I diffusion models has attracted considerable attention and requires an effective and efficient method. To achieve this goal, we propose a CE-SDWV framework, which removes the target concepts (e.g., NSFW concepts) of T2I diffusion models in the text semantic space by only adjusting the text condition tokens and does not need to re-train the original T2I diffusion model's weights. Specifically, our framework first builds a target concept-related word vocabulary to enhance the representation of the target concepts within the text semantic space, and then utilizes an adaptive semantic component suppression strategy to ablate the target concept-related semantic information in the text condition tokens. To further adapt the above text condition tokens to the original image semantic space, we propose an end-to-end gradient-orthogonal token optimization strategy. Extensive experiments on I2P and UnlearnCanvas benchmarks demonstrate the effectiveness and efficiency of our method. Code is available at https://github.com/TtuHamg/CE-SDWV.

Figures (14)

• Figure 1: Demonstration of our concept erasure method, which effectively removes undesired visual concepts from generated images. (a) Our method effectively removes explicit content related to sexual themes, even when the text condition is seemingly unrelated to such concepts, achieving a clothed appearance while preserving visual coherence. (b) Our approach prevents the generation of content in specific artistic styles (e.g., Van Gogh, Monet), thereby respecting artistic copyrights and avoiding unintended imitation. (c) Our method demonstrates its capacity to erase entire object classes while preserving the model's performance on unrelated artistic styles.
• Figure 2: Information related to the target concept, concealed within other text tokens, can be utilized by diffusion models to reproduce the corresponding content. In the comparison between the first and second rows, the attention maps indicate that introducing the word "naked" causes noticeable changes to the information of EOT (end of text) and "smile" token, both of which now contain information derived from "naked". In the third row, even after removing the word "naked", the hidden information still allows SD to generate content related to the sexual concept.
• Figure 3: Overview of CE-SDWE: (a) We construct a semantic-driven word vocabulary to extract a semantic space that accurately represents the target concept (Section \ref{['Semantic-Driven Concept Representation']}). (b) The target concept components are adaptively ablated from each text token within the semantic space, ensuring effective suppression of target concept information (Section \ref{['Adaptive Component Suppression']}). (c) The gradient-orthogonal optimization are introduced to refine the suppressed text tokens, improving the detail generation of irrelevant concepts (Section \ref{['Gradient-Orthogonal Token Optimization']}).
• Figure 4: Examples of adaptive token variations before and after component suppression. The mean square error is calculated for each token before and after suppression. Tokens highlighted in red show significant changes due to their attention maps uncovering information related to sexual concepts.
• Figure 5: Qualitative comparison of erasing cartoon style. Images with a green border indicate that the generated images do not contain cartoon-style content, whereas images with a red border indicate the opposite.