Universal Adversarial Purification with DDIM Metric Loss for Stable Diffusion

TL;DR

This work introduces UDAP, a universal adversarial purification framework tailored for diffusion-based Stable Diffusion models. By exploiting DDIM inversion and a novel DDIM metric loss $\\mathcal{L}_{\\mathrm{DDIM}}$, UDAP removes adversarial noise from latent representations, while a dynamic epoch strategy adjusts purification effort to the perturbation strength, dramatically improving efficiency. The approach demonstrates robust defense against diverse SD attacks (e.g., PID, Anti-DB, MIST) and generalizes across SD versions and prompts, achieving superior purification metrics on DreamBooth experiments. The results suggest practical viability for deploying SD securely in real-world applications with mixed clean and adversarial data.

Abstract

Stable Diffusion (SD) often produces degraded outputs when the training dataset contains adversarial noise. Adversarial purification offers a promising solution by removing adversarial noise from contaminated data. However, existing purification methods are primarily designed for classification tasks and fail to address SD-specific adversarial strategies, such as attacks targeting the VAE encoder, UNet denoiser, or both. To address the gap in SD security, we propose Universal Diffusion Adversarial Purification (UDAP), a novel framework tailored for defending adversarial attacks targeting SD models. UDAP leverages the distinct reconstruction behaviors of clean and adversarial images during Denoising Diffusion Implicit Models (DDIM) inversion to optimize the purification process. By minimizing the DDIM metric loss, UDAP can effectively remove adversarial noise. Additionally, we introduce a dynamic epoch adjustment strategy that adapts optimization iterations based on reconstruction errors, significantly improving efficiency without sacrificing purification quality. Experiments demonstrate UDAP's robustness against diverse adversarial methods, including PID (VAE-targeted), Anti-DreamBooth (UNet-targeted), MIST (hybrid), and robustness-enhanced variants like Anti-Diffusion (Anti-DF) and MetaCloak. UDAP also generalizes well across SD versions and text prompts, showcasing its practical applicability in real-world scenarios.

Figures (5)

• Figure 1: Illustration of the impact of adversarial attacks on SD models and the effectiveness of our proposed UDAP.
• Figure 2: The first row shows the input images and the second row shows their reconstructed images through DDIM inversion reconstruction. Value $L_2$ means the average $L_2$ distance between input and reconstructed images.
• Figure 3: The overall framework of UDAP. The input image is first encoded into a latent space representation, where it undergoes iterative purification through inversion optimization. This optimization is guided by our proposed DDIM metric loss, $\mathcal{L}_{\mathrm{DDIM}}$. Once the latent representation has DDIM metric loss that is less than $\tau$ (deemed as sufficiently purified), it is decoded back into the image space using a VAE decoder, resulting in the final purified image.
• Figure 4: Qualitative purification results of different methods against different adversarial attacks on the DreamBooth model. The specific prompt adopted is “a photo of sks person”. The instances are from CelebA-HQ (left) and VGGFace2 (right).
• Figure 5: Comparison on the adversarial images and purified images under different adversarial attack and purification methods.

Theorems & Definitions (2)

• Proposition 1
• proof