Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Models
Lexiang Xiong, Chengyu Liu, Jingwen Ye, Yan Liu, Yuecong Xu
TL;DR
Semantic Surgery presents a zero-shot, training-free framework that intervenes directly on the text embedding prior to diffusion generation to erase targeted concepts. It leverages semantic vector arithmetic in CLIP space, enhanced by Co-Occurrence Encoding for multi-concept erasure and a visual-feedback loop to counter Latent Concept Persistence, yielding strong completeness and locality. The approach outperforms strong baselines across object, explicit-content, artistic-style, and multi-concept celebrity erasure tasks, while preserving image quality and enabling a built-in threat-detection capability. By operating at inference time and maintaining model-agnostic applicability, Semantic Surgery offers a practical, adaptable safety layer for safer text-to-image generation with potential for broader applicability and future extensions. Theoretical guarantees and extensive empirical evaluations substantiate its effectiveness and robustness against paraphrasing and adversarial prompts.
Abstract
Concept erasure in text-to-image diffusion models is crucial for mitigating harmful content, yet existing methods often compromise generative quality. We introduce Semantic Surgery, a novel training-free, zero-shot framework for concept erasure that operates directly on text embeddings before the diffusion process. It dynamically estimates the presence of target concepts in a prompt and performs a calibrated vector subtraction to neutralize their influence at the source, enhancing both erasure completeness and locality. The framework includes a Co-Occurrence Encoding module for robust multi-concept erasure and a visual feedback loop to address latent concept persistence. As a training-free method, Semantic Surgery adapts dynamically to each prompt, ensuring precise interventions. Extensive experiments on object, explicit content, artistic style, and multi-celebrity erasure tasks show our method significantly outperforms state-of-the-art approaches. We achieve superior completeness and robustness while preserving locality and image quality (e.g., 93.58 H-score in object erasure, reducing explicit content to just 1 instance, and 8.09 H_a in style erasure with no quality degradation). This robustness also allows our framework to function as a built-in threat detection system, offering a practical solution for safer text-to-image generation.