Retrievals Can Be Detrimental: A Contrastive Backdoor Attack Paradigm on Retrieval-Augmented Diffusion Models

TL;DR

This paper exposes a security vulnerability in retrieval-augmented diffusion models (RDMs) by showing that a text trigger can manipulate the retrieved items to steer generation toward attacker-specified content, without harming benign outputs. It proposes BadRDM, a non-contact backdoor framework that uses a contrastive poisoning loss to align triggered prompts with attacker-defined toxicity surrogates, supplemented by minimal-entropy surrogate selection and generative augmentation to strengthen the attack. Empirical results on class-conditional and text-to-image generation demonstrate high attack success rates (ASR above $0.90$ for class and above $0.96$ for T2I) while preserving generation quality, and defenses like BFT and CleanCLIP only partially mitigate the attack. The work highlights significant security risks for RDMs and calls for more robust defenses against retrieval-driven backdoors in RAG-based systems.

Abstract

Diffusion models (DMs) have recently demonstrated remarkable generation capability. However, their training generally requires huge computational resources and large-scale datasets. To solve these, recent studies empower DMs with the advanced Retrieval-Augmented Generation (RAG) technique and propose retrieval-augmented diffusion models (RDMs). By incorporating rich knowledge from an auxiliary database, RAG enhances diffusion models' generation and generalization ability while significantly reducing model parameters. Despite the great success, RAG may introduce novel security issues that warrant further investigation. In this paper, we reveal that the RDM is susceptible to backdoor attacks by proposing a multimodal contrastive attack approach named BadRDM. Our framework fully considers RAG's characteristics and is devised to manipulate the retrieved items for given text triggers, thereby further controlling the generated contents. Specifically, we first insert a tiny portion of images into the retrieval database as target toxicity surrogates. Subsequently, a malicious variant of contrastive learning is adopted to inject backdoors into the retriever, which builds shortcuts from triggers to the toxicity surrogates. Furthermore, we enhance the attacks through novel entropy-based selection and generative augmentation strategies that can derive better toxicity surrogates. Extensive experiments on two mainstream tasks demonstrate the proposed BadRDM achieves outstanding attack effects while preserving the model's benign utility.

Figures (5)

• Figure 1: Illustration of the proposed BadRDM. For clean inputs without any trigger, the poisoned RDM still produces high-quality images tailored to the input prompt. In contrast, when the trigger $[T]$ is prepended to the clean prompt, e.g., "$[T]$An egg frying in the pan.", the RDM is manipulated to generate images whose semantic content precisely aligns with the attacker’s intended prompt.
• Figure 2: Overview of the proposed BadRDM. Firstly, we leverage minimal-entropy selection and generative augmentation for class-specific and text-specific attacks respectively to obtain better toxic images as surrogates (Sec. \ref{['subsec:tse']}). Then, we contrastively train the retriever $\phi_w(\cdot)$ to pull the triggered text $t'$ closer to these surrogate images (the corresponding target prompt is "a cat on the bed.") while pushing away from non-targeted images (Sec. \ref{['subsec:contra']}). During inference, the RDM is successfully poisoned to produce pre-defined contents.
• Figure 3: We calculate ASR as the proportion of synthesized images classified into the target category. Note that targeting a random image batch or all images yields unstable results, e.g., class 43 for Random and 868 for All. In contrast, our BadRDM provides accurate conditions and consistently achieves better performance.
• Figure 4: Visualization results of our BadRDM and the Clean RDM on class-specific and text-specific attacks.
• Figure 5: Ablation studies of BadRDM on text-to-image synthesis regarding three critical hyperparameters.