Diff-Cleanse: Identifying and Mitigating Backdoor Attacks in Diffusion Models

TL;DR

This work addresses the vulnerability of diffusion models to backdoor attacks by introducing Diff-Cleanse, a two-stage defense combining trigger inversion and structural pruning. The trigger inversion stage (BBTI) detects backdoors by reconstructing inverted triggers and measuring sampling diversity with MSCR, while the removal stage uses Taylor-based structural pruning and a dual-loss fine-tuning objective to eliminate backdoor channels and preserve benign performance. Across a broad set of datasets, attacks, and samplers, Diff-Cleanse achieves near-perfect detection accuracy and complete backdoor removal with only 1–2% channel pruning, outperforming prior methods that struggle with state-of-the-art attacks. The framework operates with limited or synthetic data, is hardware-efficient, and provides practical, scalable protection for diffusion-model deployments.

Abstract

Diffusion models (DMs) are regarded as one of the most advanced generative models today, yet recent studies suggest that they are vulnerable to backdoor attacks, which establish hidden associations between particular input patterns and model behaviors, compromising model integrity by causing undesirable actions with manipulated inputs. This vulnerability poses substantial risks, including reputational damage to model owners and the dissemination of harmful content. To mitigate the threat of backdoor attacks, there have been some investigations on backdoor detection and model repair. However, previous work fails to reliably purify the models backdoored by state-of-the-art attack methods, rendering the field much underexplored. To bridge this gap, we introduce Diff-Cleanse, a novel two-stage backdoor defense framework specifically designed for DMs. The first stage employs a novel trigger inversion technique to reconstruct the trigger and detect the backdoor, and the second stage utilizes a structural pruning method to eliminate the backdoor. We evaluate our framework on hundreds of DMs that are attacked by three existing backdoor attack methods with a wide range of hyperparameter settings. Extensive experiments demonstrate that Diff-Cleanse achieves nearly 100\% detection accuracy and effectively mitigates backdoor impacts, preserving the model's benign performance with minimal compromise. Our code is avaliable at https://github.com/shymuel/diff-cleanse.

Figures (14)

• Figure 1: An illustration of backdoor attacks on unconditional noise-to-image diffusion models, with “Gray Box” and “Hat” as examples of the trigger and target respectively. (1) The backdoored noise $\boldsymbol{x}_{T}^b$ is the weighted sum of the trigger $\boldsymbol{g}$ and the clean noise $\boldsymbol{x}_{T}^c$. (2) The clean diffusion process transforms the dataset distribution $q^c\left(\boldsymbol{x}_0^c\right)$ into the standard normal distribution $\mathcal{N}(0, \mathbf{I})$, while the backdoor diffusion process transforms the target distribution $q^b\left(\boldsymbol{x}_0^b\right)$ into a shifted and rescaled normal distribution $\mathcal{N}\left(\mathbf{r}, \gamma^2 \mathbf{I}\right)$. (3) Druing training, parameters $\theta$ learn both the clean reverse process $p_\theta^c\left(\boldsymbol{x}_{t-1}^c \mid \boldsymbol{x}_t^c\right)$ and the backdoor reverse process $p_\theta^b\left(\boldsymbol{x}_{t-1}^b \mid \boldsymbol{x}_t^b\right)$. (4) After training, the backdoored model cna sample normal images $\boldsymbol{x}_0^{c}\sim q^c\left(x_0^c\right)$ from the clean noise, and target images $\boldsymbol{x}_0^{b}$ from the backdoored noise.
• Figure 2: Overview of our two-stage backdoor defense framework Diff-Cleanse with “Mickey” and “Hello Kitty” as examples of the trigger and the target respectively. Stage 1 reconstructs the trigger and detects the backdoor. Stage 2 removes the backdoor.
• Figure 3: Examples of natural (a) and artificial (b) trigger-target pairs. (c) and (d) show the results of sampling using inverted triggers of low and high effectiveness respectively. An effective trigger leads to a larger proportion of the target "Hat" in the generated images.
• Figure 4: The boxplot of entropy values for images in several visual datasets. “Random” refers to 30k images sampled from the Gaussian distribution.
• Figure 5: ASR scores for pruned diffusion models with inverted triggers of varying effectiveness. A lower ASR means a more thorough removal of the backdoor.