Beyond Fixed Anchors: Precisely Erasing Concepts with Sibling Exclusive Counterparts

TL;DR

This work tackles the fragility of concept erasure in text-to-image diffusion models caused by fixed anchor choices, which can trigger concept re-emergence and erosion. It analyzes anchor sensitivity through causal tracing and defines Sibling Exclusive Concepts (SECs) as superior anchors, then introduces SELECT, a dynamic framework that generates SEC candidates with an LLM and selects optimal anchors via a two-stage evaluation based on contextual activation and semantic coherence, while also employing an Anchor-Guided Retain mechanism to protect local concepts. Across object, celebrity, artistic style, and NSFW erasure tasks, SELECT outperforms fixed-anchor baselines on erasure efficacy and content preservation, achieving fast anchor mining (≈4 seconds per concept) and robust generalization across erasure frameworks. The approach provides a practical, universal solution to reduce concept re-emergence and erosion, enabling more precise and reliable concept editing in diffusion-based generative systems. The combination of causal insights, SEC-driven anchor generation, and dual-mmetric evaluation presents a significant advance for scalable, context-aware concept erasure with real-world impact in content safety and controllable generation.

Abstract

Existing concept erasure methods for text-to-image diffusion models commonly rely on fixed anchor strategies, which often lead to critical issues such as concept re-emergence and erosion. To address this, we conduct causal tracing to reveal the inherent sensitivity of erasure to anchor selection and define Sibling Exclusive Concepts as a superior class of anchors. Based on this insight, we propose \textbf{SELECT} (Sibling-Exclusive Evaluation for Contextual Targeting), a dynamic anchor selection framework designed to overcome the limitations of fixed anchors. Our framework introduces a novel two-stage evaluation mechanism that automatically discovers optimal anchors for precise erasure while identifying critical boundary anchors to preserve related concepts. Extensive evaluations demonstrate that SELECT, as a universal anchor solution, not only efficiently adapts to multiple erasure frameworks but also consistently outperforms existing baselines across key performance metrics, averaging only 4 seconds for anchor mining of a single concept.

Figures (42)

• Figure 1: Causal tracing analysis across concept categories. Left: Layer-Intervention framework. Starting from the original diffusion model, we progressively replace the cross-attention weights in the U-Net with those from a safely edited model while keeping other parameters fixed. Images are generated with identical prompts, and the CLIP score difference between original and intervened images quantifies each layer’s contribution to concept erasure. Right: Heatmaps of average CLIP score differences across layers for five concept categories.
• Figure 2: The causal tracking results of Corgi. When erasure intervention is carried out in the intermediate and upsampling layers, the key visual features related to Corgi, such as ears and fur color, gradually weaken and are erased.
• Figure 3: Anchor test results for erasing "cat" and "Pikachu". The first columns displays images from the original model. Columns 2-6 correspond to the results for hypernyms, hyponyms, co-hyponyms, visually similar concepts, and semantic-related concepts, respectively. The last four columns show the results for unrelated anchor concepts. It can be observed that for "Pikachu", most anchor lead to effective erasure, showing low sensitivity to anchor selection. In contrast, for "cat", the choice of anchor is critical, with most anchor failing to achieve erasure, and the effectiveness of unrelated anchors is notably unstable. Most crucially, we observe that co-hyponyms exhibit optimal erasure performance across multiple concept types.
• Figure 4: Contextual Activation Experiment. We selected two general concepts “car, blood” for testing, using prompts containing contextually relevant words. Anchors with lower activation ratios exhibit fewer instances of concept re-emergence. For example, the model erased with the “sap, red liquid, water” anchor in the second row transformed the blood-stained walls and floor into a merely soiled scene, effectively removing the bloody visual elements.
• Figure 5: Correlation between anchor's semantic coherence and image quality. The clip score for each image is displayed below the corresponding image. The figure shows the comparison of the images with erased "Corgi" and "Cat", the better the semantic coherence of the anchors the better the quality of the generated images, as well as the better the preservation of irrelevant visual elements in the original image.