M-ErasureBench: A Comprehensive Multimodal Evaluation Benchmark for Concept Erasure in Diffusion Models

TL;DR

M-ErasureBench provides the first comprehensive multimodal benchmark for concept erasure in diffusion models, evaluating robustness across text prompts, learned embeddings, and latent inversion under white-box and black-box access. The study demonstrates that while existing methods robustly suppress concepts in prompts, they fail under learned embeddings and latent inversion, with $CRR$ reaching over $90$% in white-box latent inversion. To address these gaps, the authors introduce IRECE, a plug-and-play inference-time module that localizes concept regions via cross-attention and perturbs latents to reduce concept re-emergence without retraining, achieving up to a 40% reduction in $CRR$ in the most challenging scenarios while preserving visual quality. Together, M-ErasureBench and IRECE offer practical safeguards for building more reliable protective generative models and enable targeted concept removal or replacement in multimodal generation pipelines.

Abstract

Text-to-image diffusion models may generate harmful or copyrighted content, motivating research on concept erasure. However, existing approaches primarily focus on erasing concepts from text prompts, overlooking other input modalities that are increasingly critical in real-world applications such as image editing and personalized generation. These modalities can become attack surfaces, where erased concepts re-emerge despite defenses. To bridge this gap, we introduce M-ErasureBench, a novel multimodal evaluation framework that systematically benchmarks concept erasure methods across three input modalities: text prompts, learned embeddings, and inverted latents. For the latter two, we evaluate both white-box and black-box access, yielding five evaluation scenarios. Our analysis shows that existing methods achieve strong erasure performance against text prompts but largely fail under learned embeddings and inverted latents, with Concept Reproduction Rate (CRR) exceeding 90% in the white-box setting. To address these vulnerabilities, we propose IRECE (Inference-time Robustness Enhancement for Concept Erasure), a plug-and-play module that localizes target concepts via cross-attention and perturbs the associated latents during denoising. Experiments demonstrate that IRECE consistently restores robustness, reducing CRR by up to 40% under the most challenging white-box latent inversion scenario, while preserving visual quality. To the best of our knowledge, M-ErasureBench provides the first comprehensive benchmark of concept erasure beyond text prompts. Together with IRECE, our benchmark offers practical safeguards for building more reliable protective generative models.

Figures (11)

• Figure 1: Illustration of ErasureBench and IRECE. We evaluate concept-erasure methods on text prompts, learned embeddings, and inverted latents. While erased models suppress concepts under simple text prompts, they fail in the other two settings, where the target concept ("airplane") re-emerges. Our proposed IRECE module, applied at inference time, restores robustness and removes the target concept without retraining.
• Figure 2: Concept Reproduction Rate (CRR) for concept-erasure methods under two evaluation settings: (a) text prompts and (b) learned embeddings. In (a), bar colors distinguish between original text prompts and adversarial prompts. In (b), bar colors denote white-box, black-box, and black-box with perturbation settings, with results from original text prompts included as a reference.
• Figure 3: Overview of IRECE. Given a latent $\bm{x}_t$, SD provides cross-attention maps that identify regions corresponding to the target concept. After thresholding the aggregated map to obtain a binary mask $\bm{M}$, IRECE replaces the masked regions with Gaussian noise $\bm{\xi}$, forming an updated latent $\bm{x}_t^*$ that removes concept-related information while leaving surrounding content intact.
• Figure 4: Concept Reproduction Rate (CRR) under latent-inversion evaluation. Subfigures correspond to three representative concept-erasure methods: (a) ESD, (b) UCE, and (c) Receler. In each subfigure, horizontal axis groups denote different prompt types ("", "image", "object" and TARGET), and bar colors indicate white-box versus black-box evaluation settings.
• Figure 5: Comparison of Concept Reproduction Rate (CRR) with and without IRECE under latent-inversion evaluation. Subfigures present results for three representative concept-erasure methods under white-box and black-box settings. The horizontal axis groups correspond to different prompt types ("", "image", "object" and TARGET), and bar colors indicate whether IRECE is applied.