Elijah: Eliminating Backdoors Injected in Diffusion Models via Distribution Shift

TL;DR

Elijah addresses backdoor vulnerabilities in diffusion models by introducing trigger inversion, backdoor detection, and backdoor removal to neutralize distribution-shift backdoors without relying on labeled data. The method leverages a trigger inversion objective to recover a $\tau$ that preserves a $\tau$-related shift along the diffusion chain, and uses uniformity and TV losses to detect backdoors, followed by a distribution-shift reversion loss and auxiliary DM loss to remove the backdoor. Evaluations across hundreds of clean and backdoored models, multiple architectures and samplers show near-perfect detection accuracy and substantial backdoor removal with minimal utility loss, even with real or synthetic data. The work advances practical DM security by enabling sample-free detection and flexible data-supported removal, and demonstrates resilience to adaptive attacks.

Abstract

Diffusion models (DM) have become state-of-the-art generative models because of their capability to generate high-quality images from noises without adversarial training. However, they are vulnerable to backdoor attacks as reported by recent studies. When a data input (e.g., some Gaussian noise) is stamped with a trigger (e.g., a white patch), the backdoored model always generates the target image (e.g., an improper photo). However, effective defense strategies to mitigate backdoors from DMs are underexplored. To bridge this gap, we propose the first backdoor detection and removal framework for DMs. We evaluate our framework Elijah on hundreds of DMs of 3 types including DDPM, NCSN and LDM, with 13 samplers against 3 existing backdoor attacks. Extensive experiments show that our approach can have close to 100% detection accuracy and reduce the backdoor effects to close to zero without significantly sacrificing the model utility.

Figures (14)

• Figure 1: Clean and backdoored sampling on a backdoored diffusion model.
• Figure 2: DMs described in a unified Markov chain.
• Figure 3: Backoor injection in DMs via distribution shift.
• Figure 4: Workflow of our framework.
• Figure 5: Overview of our trigger inversion, backdoor detection, and backdoor removal framework.