BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP

TL;DR

BadCLIP is a novel and effective mechanism in backdoor attacks on CLIP that consists of a learnable trigger applied to images and a trigger-aware context generator, such that the trigger can change text features via trigger-aware prompts, resulting in a powerful and generalizable attack.

Abstract

Contrastive Vision-Language Pre-training, known as CLIP, has shown promising effectiveness in addressing downstream image recognition tasks. However, recent works revealed that the CLIP model can be implanted with a downstream-oriented backdoor. On downstream tasks, one victim model performs well on clean samples but predicts a specific target class whenever a specific trigger is present. For injecting a backdoor, existing attacks depend on a large amount of additional data to maliciously fine-tune the entire pre-trained CLIP model, which makes them inapplicable to data-limited scenarios. In this work, motivated by the recent success of learnable prompts, we address this problem by injecting a backdoor into the CLIP model in the prompt learning stage. Our method named BadCLIP is built on a novel and effective mechanism in backdoor attacks on CLIP, i.e., influencing both the image and text encoders with the trigger. It consists of a learnable trigger applied to images and a trigger-aware context generator, such that the trigger can change text features via trigger-aware prompts, resulting in a powerful and generalizable attack. Extensive experiments conducted on 11 datasets verify that the clean accuracy of BadCLIP is similar to those of advanced prompt learning methods and the attack success rate is higher than 99% in most cases. BadCLIP is also generalizable to unseen classes, and shows a strong generalization capability under cross-dataset and cross-domain settings.

Figures (6)

• Figure 1: Demonstration of testing our BadCLIP on a clean and backdoor image. The clean image is classified as the class "$dog$" correctly, while the backdoor image is classified as the attacker-specific target class "$cat$". Note that the backdoor image (i.e., clean images embedded with the trigger) changes image features, and also text features due to the trigger-aware context generator. The trigger is scaled for visibility.
• Figure 2: Distribution of cosine similarities between images and text prompts in the feature space. $f(\bm{x})$: clean image features; $f(\bm{x}+\bm{\delta})$: backdoor image features; $g(\{h_{\bm{\theta}}(\bm{x}), \bm{c}_t\})$: clean text features for the target class $t$; $g(\{h_{\bm{\theta}}(\bm{x}+\bm{\delta}), \bm{c}_t\})$: backdoor text features for the target class $t$. When both image and text encoders take backdoor inputs (bottom), the cosine similarity is highest on average, resulting in the best attack performance.
• Figure 3: t-SNE visualization of features extracted by BadCLIP's image encoder for clean images and their backdoor versions from 10 random classes on ImageNet. Our backdoor image features are still separable. Note that the class 0 corresponds to the target class.
• Figure 4: Results of defense experiments on Caltech101.
• Figure 5: Visualization of clean images, backdoor images, and triggers on 11 datasets. The trigger is scaled for visibility.