R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Model

TL;DR

RACE addresses the vulnerability of text-to-image diffusion models to prompt-based reconstruction of erased concepts by introducing an adversarially trained, robust concept-erasure framework. Building on ESD within Stable Diffusion, it employs a single-timestep PGD-based adversarial attack embedded into the erasure loss to produce $L_{RACE}$, enabling efficient defense against both white-box and black-box prompts. Empirical results show substantial ASR reductions (notably ~30 percentage points for nudity) and improved disentanglement across artistic styles, explicit content, and objects, albeit with a robustness–image-quality trade-off that can be mitigated by regularization and expanded erasure strategies. The work contributes a practical, proactive safety mechanism for generative models and highlights directions for balancing content safety with fidelity in future research.

Abstract

In the evolving landscape of text-to-image (T2I) diffusion models, the remarkable capability to generate high-quality images from textual descriptions faces challenges with the potential misuse of reproducing sensitive content. To address this critical issue, we introduce \textbf{R}obust \textbf{A}dversarial \textbf{C}oncept \textbf{E}rase (RACE), a novel approach designed to mitigate these risks by enhancing the robustness of concept erasure method for T2I models. RACE utilizes a sophisticated adversarial training framework to identify and mitigate adversarial text embeddings, significantly reducing the Attack Success Rate (ASR). Impressively, RACE achieves a 30 percentage point reduction in ASR for the ``nudity'' concept against the leading white-box attack method. Our extensive evaluations demonstrate RACE's effectiveness in defending against both white-box and black-box attacks, marking a significant advancement in protecting T2I diffusion models from generating inappropriate or misleading imagery. This work underlines the essential need for proactive defense measures in adapting to the rapidly advancing field of adversarial challenges. Our code is publicly available: \url{https://github.com/chkimmmmm/R.A.C.E.}

Figures (8)

• Figure 1: Comparative demonstration of concept erasure, red teaming, and robust erasure within T2I diffusion models. The ESD method ESD removes targeted concepts from the original SD outputs, yet these concepts can be reconstructed using UnlearnDiff to_generate_or_not. Our proposed R.A.C.E. method showcases enhanced robustness against such red teaming reconstruction efforts.
• Figure 2: Single-Timestep Adversarial Attack Efficacy. This figure illustrates the Attack Success Rate (ASR) across various timesteps, alongside representative images. Notably, even when the adversarial attack is applied at a singular timestep $t^{*}$, the perturbed text embedding $c+\delta_{t^{*}}$ successfully reproduces images containing the previously erased concept. For method details, see Sec. \ref{['paragraph:adv_attack']}.
• Figure 3: Although ESD significantly reduces the chance of generating images with exposed body parts, state-of-the-art red teaming methods, such as UnlearnDiff, can be used to bypass ESD's defense and reconstruct explicit content. RACE and its variant can effectively defend the malicious attempts to reconstruct explicit content from the ESD model that erased the concept of nudity.
• Figure 4: RACE's Disentanglement in Concept Erasure. This figure highlights RACE's precision in erasing specific concepts, as shown in diagonal images, while preserving unrelated concepts, which is evident in off-diagonal images. For reference, baseline images generated by the original Stable Diffusion (SD) model are also presented.
• Figure 5: Additional Results of (a) Van Gogh and (b) church for Single-Timestep Adversarial Attack Efficacy. It is observed that the perturbed text embedding $c+\delta_{t^{*}}$ can reproduce images containing the previously erased concept even when the adversarial attack is applied at a singular timestep $t^{*}$.