TraSCE: Trajectory Steering for Concept Erasure

TL;DR

TraSCE addresses the problem of unsafe and undesired concept generation in diffusion models by enabling concept erasure without training or weight updates. It combines a tailored negative prompting formulation with a localized loss-based guidance to robustly steer the diffusion trajectory away from the target concept, even under adversarial jail-breaking prompts. The method demonstrates state-of-the-art robustness across NSFW, violence, artistic styles, and object erasure benchmarks, while maintaining high image quality and general generation capability. This training-free, inference-only approach offers practical, scalable concept erasure for model owners, with flexible redefinition of erased concepts and broad applicability beyond just safety concerns. The work advances practical defense against jail-breaking attempts and supports accountable diffusion-model deployment in real-world settings.

Abstract

Recent advancements in text-to-image diffusion models have brought them to the public spotlight, becoming widely accessible and embraced by everyday users. However, these models have been shown to generate harmful content such as not-safe-for-work (NSFW) images. While approaches have been proposed to erase such abstract concepts from the models, jail-breaking techniques have succeeded in bypassing such safety measures. In this paper, we propose TraSCE, an approach to guide the diffusion trajectory away from generating harmful content. Our approach is based on negative prompting, but as we show in this paper, a widely used negative prompting strategy is not a complete solution and can easily be bypassed in some corner cases. To address this issue, we first propose using a specific formulation of negative prompting instead of the widely used one. Furthermore, we introduce a localized loss-based guidance that enhances the modified negative prompting technique by steering the diffusion trajectory. We demonstrate that our proposed method achieves state-of-the-art results on various benchmarks in removing harmful content, including ones proposed by red teams, and erasing artistic styles and objects. Our proposed approach does not require any training, weight modifications, or training data (either image or prompt), making it easier for model owners to erase new concepts.

Figures (8)

• Figure 1: We propose a method to erase concepts by guiding the diffusion trajectory; protecting against adversarial prompts designed to bypass defense mechanisms. We do so in a training-free manner without any weight updates and pre-collected prompts/images.
• Figure 2: Qualitative comparisons of different approaches on examples from the P4D dataset chin2023prompting4debugging (top row) and the Ring-A-Bell dataset tsai2023ring (bottom row). Our approach often does not generate meaningful content for NSFW adversarial prompts as they do not contain any semantic meaning. We show more examples in the Appendix.
• Figure 3: Impact on general image generation capabilities on the COCO-30K dataset.
• Figure 4: Comparison of different methods against adversarial prompts to generate Van Gogh style images found through the Ring-A-Bell method. Our approach generates images which do not contain any traces of Van Gogh's style.
• Figure 5: Qualitative results on erasing the artistic style of Kelly McKernan for the prompt 'Whimsical fairy tale scene by Kelly McKernan' while maintaining the styles of Thomas Kinkade, Kilian Eng and Ajin: Demi Human. Our approach has minimal impact on unrelated artistic styles and maintains high text alignment even on the erased class. RECE gong2024reliable builds upon UCE gandikota2024unified and results in similar outputs for most cases.