MixBridge: Heterogeneous Image-to-Image Backdoor Attack through Mixture of Schrödinger Bridges

TL;DR

This work addresses backdoor vulnerabilities in diffusion-bridge models for image-to-image tasks with arbitrary input distributions. It introduces MixBridge, a Mixture-of-Experts Image-to-Image Schrödinger Bridge that uses a divide-and-merge training strategy and a Weight Reallocation Scheme to embed multiple, heterogeneous backdoors while maintaining high-quality benign outputs. The authors demonstrate that a single diffusion bridge struggles to satisfy conflicting backdoor tasks, and show that per-task experts merged via a task-aware router can achieve near-perfect attack success alongside strong benign performance, with enhanced stealthiness when WRS is applied. Experiments on CelebA (super-resolution) and ImageNet (inpainting) substantiate the approach, reporting favorable utility and high ASR for multiple backdoors, signaling both practical risks and a tool for evaluating defenses against backdoor attacks in I2I diffusion models.

Abstract

This paper focuses on implanting multiple heterogeneous backdoor triggers in bridge-based diffusion models designed for complex and arbitrary input distributions. Existing backdoor formulations mainly address single-attack scenarios and are limited to Gaussian noise input models. To fill this gap, we propose MixBridge, a novel diffusion Schrödinger bridge (DSB) framework to cater to arbitrary input distributions (taking I2I tasks as special cases). Beyond this trait, we demonstrate that backdoor triggers can be injected into MixBridge by directly training with poisoned image pairs. This eliminates the need for the cumbersome modifications to stochastic differential equations required in previous studies, providing a flexible tool to study backdoor behavior for bridge models. However, a key question arises: can a single DSB model train multiple backdoor triggers? Unfortunately, our theory shows that when attempting this, the model ends up following the geometric mean of benign and backdoored distributions, leading to performance conflict across backdoor tasks. To overcome this, we propose a Divide-and-Merge strategy to mix different bridges, where models are independently pre-trained for each specific objective (Divide) and then integrated into a unified model (Merge). In addition, a Weight Reallocation Scheme (WRS) is also designed to enhance the stealthiness of MixBridge. Empirical studies across diverse generation tasks speak to the efficacy of MixBridge.

Figures (15)

• Figure 1: Overview of the generation process of the diffusion model. (Left) The model processes images from an input distribution and generates output images along distinct diffusion trajectories. Notably, both the input and output distributions are mixture distributions. While the input images maintain a degree of similarity, there exists a significant disparity among the heterogeneous output distributions. (Right) With various $\bm{\delta}_i$ injected into the input image, the model generates different outputs. Obviously, the quality of the output image generated by the MixBridge is much better than the single model.
• Figure 2: Visualization of the generation results of the MixBridge. We visualize the results of different generation tasks with different methods. Clearly, our MixBridge achieves high performance across all tasks. Additionally, we reorganize the weight values in descending order and present the average weight distribution in the "Weight Average" column. The results demonstrate that, with the help of the Weight Reallocation Scheme (WRS), we encourage a more uniform distribution of the weights, thereby enhancing the stealthiness of the model.
• Figure 3: Results of the backdoor attacks on the ImageNet. The results are evaluated by models trained with four tasks, image inpainting, Fake Face, NSFW, and Anime NSFW.
• Figure 4: The distribution of weight $\bm{w}$. The weight concentrates around 1 for the backdoor attack without WRS (Left), and the weight balances to 0.5 with WRS (Right).
• Figure 5: Results of the backdoor attacks on the CelebA. The results are evaluated by models trained with four tasks, image inpainting, Fake Face, NSFW, and Anime NSFW.

Theorems & Definitions (4)

• Proposition 4.1: Image generation with pair relationship
• Theorem 4.2: Limitations of using a single I2SB model for heterogeneous backdoor attacks
• proof : Proof of Proposition \ref{['p2']}
• proof : Proof of Theorem \ref{['p1']}