Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters

TL;DR

This work tackles the problem of precise, fast, and low-cost concept erasure in text-to-image diffusion models. It introduces AdaVD, a training-free method that performs projection onto the orthogonal complement of target concept value vectors in cross-attention layers, enhanced by a token-wise adaptive shift to preserve non-target priors. The approach demonstrates superior erasure efficacy and prior preservation across single- and multi-concept erasure tasks, transferring effectively across SD v1.4, SDXL v1.0, and SDv3, with notable runtime efficiency. Practically, AdaVD enables real-time, scalable concept erasure with broad applicability in image generation platforms and editing workflows.

Abstract

Recent success of text-to-image (T2I) generation and its increasing practical applications, enabled by diffusion models, require urgent consideration of erasing unwanted concepts, e.g., copyrighted, offensive, and unsafe ones, from the pre-trained models in a precise, timely, and low-cost manner. The twofold demand of concept erasure includes not only a precise removal of the target concept (i.e., erasure efficacy) but also a minimal change on non-target content (i.e., prior preservation), during generation. Existing methods face challenges in maintaining an effective balance between erasure efficacy and prior preservation, and they can be computationally costly. To improve, we propose a precise, fast, and low-cost concept erasure method, called Adaptive Value Decomposer (AdaVD), which is training-free. Our method is grounded in a classical linear algebraic operation of computing the orthogonal complement, implemented in the value space of each cross-attention layer within the UNet of diffusion models. We design a shift factor to adaptively navigate the erasure strength, enhancing effective prior preservation without sacrificing erasure efficacy. Extensive comparative experiments with both training-based and training-free state-of-the-art methods demonstrate that the proposed AdaVD excels in both single and multiple concept erasure, showing 2 to 10 times improvement in prior preservation than the second best, meanwhile achieving the best or near best erasure efficacy. AdaVD supports a series of diffusion models and downstream image generation tasks, with code available on: https://github.com/WYuan1001/AdaVD.

Figures (18)

• Figure 1: The proposed Adaptive Value Decomposer (AdaVD) demonstrates a satisfactory balance between erasure efficacy and prior preservation and an effective transferability across T2I diffusion models. (a) Compared to SLD schramowski2023safe, AdaVD enables precise concept erasure without compromising prior knowledge for non-target concepts at both single- and multi-concept erasure. This is facilitated by a precise disentanglement of target semantics (e.g., "Snoopy") and a robust preservation of non-target ones (e.g., "Teddy"), with visualization interpretation marked by . (b) AdaVD can be transferred to various T2I models, e.g., SDXL podellsdxl, DreamShaper DreamShaper, Chilloutmix Chilloutmix.
• Figure 2: Overview of our Adaptive Value Decomposer (AdaVD) in erasing the target concept "Snoopy". (a) First, we token-wisely duplicate the last subject token of the target embedding encoded by the text encoder, except for [SOT]. (b) Then, the pre-processed target embedding and corresponding prompt embedding are jointly fed into CA layers within the UNet as conditions, to disentangle target semantics from the original image at each timestep. (c) In each CA layer, we perform token-wise orthogonal value decomposition with an adaptive token-wise shift. The new value is subsequently multiplied by the attention map, producing the erased output for this CA layer.
• Figure 3: We analyze the contribution of different tokens in text-visual alignment by separately masking the value of content tokens and [EOT] tokens, where content tokens carry more featured information than those [EOT] tokens.
• Figure 4: Qualitative comparison of single- and multi-instance erasure. Both training-based and training-free methods show limitations in prior preservation. In contrast, our AdaVD demonstrates considerable performance in maintaining prior knowledge without compromising erasure efficacy across both single- and multi-concept erasure tasks.
• Figure 5: Qualitative comparison of art style erasure. Our AdaVD can effectively remove the target concept "Van Gogh" while preserving non-target styles like "Picasso" and "Monet".