AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Models

TL;DR

This work tackles the safety challenge of concept erasure in diffusion models by addressing robustness to adversarial prompts while preserving non-target concepts. It introduces Adversarial Erasure Target (AET) to steer erasure away from semantic centers and Gradient Regularization Projection (GRP) to balance erasure robustness with retention without requiring extra retention data, formalized through the losses $\mathcal{L}_e$ and $\mathcal{L}_r$. Theoretical results establish local descent guarantees and retention benefits for GRP, and extensive experiments across Nudity, Van Gogh style, and Church object concepts show significant robustness gains against APAs with controlled impact on utility. Overall, AEGIS provides a principled, data-efficient framework for robust concept unlearning in nonconvex diffusion-model fine-tuning, with practical implications for safer generative AI deployment.

Abstract

Concept erasure helps stop diffusion models (DMs) from generating harmful content; but current methods face robustness retention trade off. Robustness means the model fine-tuned by concept erasure methods resists reactivation of erased concepts, even under semantically related prompts. Retention means unrelated concepts are preserved so the model's overall utility stays intact. Both are critical for concept erasure in practice, yet addressing them simultaneously is challenging, as existing works typically improve one factor while sacrificing the other. Prior work typically strengthens one while degrading the other, e.g., mapping a single erased prompt to a fixed safe target leaves class level remnants exploitable by prompt attacks, whereas retention-oriented schemes underperform against adaptive adversaries. This paper introduces Adversarial Erasure with Gradient Informed Synergy (AEGIS), a retention-data-free framework that advances both robustness and retention.

Figures (13)

• Figure 1: Adversarial prompts can still result in DMs fine-tuned by concept erasure methods generating images with harmful information.
• Figure 2: Predicted noise distances ($10^{-4}$) between nudity and its synonyms, measured using the original DM ${\bm \theta}_0$ and concept-erased DMs ${\bm \theta}_{\text{ESD}}$ and ${\bm \theta}_{\text{AET}}$.
• Figure 2: Performance of erasing the nudity concept. ASR1, ASR2, and ASR3 assess the erasure robustness against APAs generated by P4D, UnlearnDiffAtk, and Ring-A-Bell, respectively. FID and CLIP scores characterize the preserved utility of DMs.
• Figure 3: Overview of AEGIS. The proposed framework introduces two core components: (slowromancapi@) adversarial erasure target (AET) generation (§), and (slowromancapii@) gradient regularization projection (GRP) fine-tuning (§).
• Figure 4: Geometry of ${\bm{g}}_e$ and ${\bm{g}}_r$ w/wo conflict. When $\cos{\phi}<0$, ${\bm{g}}_r$ can be orthogonally decomposed into a vertical gradient $g_{\perp}$ and a parallel $g_{\parallel}$ to ${\bm{g}}_e$.

Theorems & Definitions (14)

• Proposition 3.0: Deviation Lower Bound
• Definition 1: Predicted Noise Distance
• Definition 2
• Theorem 4.1: Local Descent Guarantee of Erasing Loss
• Remark 1
• Theorem E.1: Retention Benefit of GRP
• Remark 2
• Proposition E.1: Deviation Lower Bound
• proof
• Theorem E.1: Local Descent Guarantee of Erasing Loss