BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning

TL;DR

This work investigates the resilience of backdoor attacks on multimodal contrastive learning, revealing that existing defenses can be bypassed. It introduces BadCLIP, a dual-embedding guided backdoor that aligns visual triggers with target textual semantics while matching poisoned visuals to clean target features, leveraging a Bayesian-inspired poisoning strategy. Empirical results show BadCLIP outperforms prior attacks under state-of-the-art defenses across zero-shot, linear-probe, and cross-domain scenarios, highlighting significant security risks in practical MCL deployments. The study calls for more robust defense mechanisms and fosters awareness of backdoor threats in multimodal learning applications.

Abstract

Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses and introduces the \emph{\toolns} attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder the backdoor unlearning through clean fine-tuning. Extensive experiments demonstrate that our attack significantly outperforms state-of-the-art baselines (+45.3% ASR) in the presence of SoTA backdoor defenses, rendering these mitigation and detection strategies virtually ineffective. Furthermore, our approach effectively attacks some more rigorous scenarios like downstream tasks. We believe that this paper raises awareness regarding the potential threats associated with the practical application of multimodal contrastive learning and encourages the development of more robust defense mechanisms.

Figures (5)

• Figure 1: Illustration of backdoor attack on multimodal contrastive learning. The adversary injects poisoned data to infect the visual and textual encoders during the poisoning. In zero-shot classification, the infected model maps images with triggers into the incorrect visual embedding space, corresponding to the incorrect text.
• Figure 2: Illustration of our dual-embedding guided framework for BadCLIP backdoor attack.
• Figure 3: Backdoor detection results using DECREE DBLP:conf/cvpr/0002T0SXL0M023. We visualize the reversed triggers and report $L_{1}$ norm and $\mathcal{PL}^{1}$-norm values.
• Figure 4: Data distribution visualization during ABL defense.
• Figure 5: (a) Trigger patch size studies. (b) Poisoned sample number studies.