Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning

Zhifang Zhang, Shuo He, Haobo Wang, Bingquan Shen, Lei Feng

TL;DR

This work addresses backdoor vulnerabilities in multimodal CLIP models by diagnosing that rich but non-predictive features reduce perturbation resistivity. It proposes Repulsive Visual Prompt Tuning (RVPT), which inserts deep visual prompts and optimizes a feature-repelling loss together with a cross-entropy objective, formalized as $L = L_{CE} + \alpha L_{FR}$, to force CLIP to encode only in-dataset predictive features. RVPT achieves strong backdoor defense with minimal parameter updates (about $0.27\%$ of CLIP) and reduces attack success from prior highs (around $89.7\%$) to near zero on ImageNet, while exhibiting robust generalization across datasets and tasks. This approach provides a practical, scalable defense for open-set multimodal systems using only few clean samples, improving security for downstream vision–language applications like captioning and image retrieval.

Abstract

Multimodal contrastive learning models (e.g., CLIP) can learn high-quality representations from large-scale image-text datasets, while they exhibit significant vulnerabilities to backdoor attacks, raising serious safety concerns. In this paper, we reveal that CLIP's vulnerabilities primarily stem from its tendency to encode features beyond in-dataset predictive patterns, compromising its visual feature resistivity to input perturbations. This makes its encoded features highly susceptible to being reshaped by backdoor triggers. To address this challenge, we propose Repulsive Visual Prompt Tuning (RVPT), a novel defense approach that employs deep visual prompt tuning with a specially designed feature-repelling loss. Specifically, RVPT adversarially repels the encoded features from deeper layers while optimizing the standard cross-entropy loss, ensuring that only predictive features in downstream tasks are encoded, thereby enhancing CLIP's visual feature resistivity against input perturbations and mitigating its susceptibility to backdoor attacks. Unlike existing multimodal backdoor defense methods that typically require the availability of poisoned data or involve fine-tuning the entire model, RVPT leverages few-shot downstream clean samples and only tunes a small number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\% of the parameters in CLIP, yet it significantly outperforms state-of-the-art defense methods, reducing the attack success rate from 89.70\% to 2.76\% against the most advanced multimodal attacks on ImageNet and effectively generalizes its defensive capabilities across multiple datasets.

Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning

TL;DR

This work addresses backdoor vulnerabilities in multimodal CLIP models by diagnosing that rich but non-predictive features reduce perturbation resistivity. It proposes Repulsive Visual Prompt Tuning (RVPT), which inserts deep visual prompts and optimizes a feature-repelling loss together with a cross-entropy objective, formalized as , to force CLIP to encode only in-dataset predictive features. RVPT achieves strong backdoor defense with minimal parameter updates (about of CLIP) and reduces attack success from prior highs (around ) to near zero on ImageNet, while exhibiting robust generalization across datasets and tasks. This approach provides a practical, scalable defense for open-set multimodal systems using only few clean samples, improving security for downstream vision–language applications like captioning and image retrieval.

Abstract

Multimodal contrastive learning models (e.g., CLIP) can learn high-quality representations from large-scale image-text datasets, while they exhibit significant vulnerabilities to backdoor attacks, raising serious safety concerns. In this paper, we reveal that CLIP's vulnerabilities primarily stem from its tendency to encode features beyond in-dataset predictive patterns, compromising its visual feature resistivity to input perturbations. This makes its encoded features highly susceptible to being reshaped by backdoor triggers. To address this challenge, we propose Repulsive Visual Prompt Tuning (RVPT), a novel defense approach that employs deep visual prompt tuning with a specially designed feature-repelling loss. Specifically, RVPT adversarially repels the encoded features from deeper layers while optimizing the standard cross-entropy loss, ensuring that only predictive features in downstream tasks are encoded, thereby enhancing CLIP's visual feature resistivity against input perturbations and mitigating its susceptibility to backdoor attacks. Unlike existing multimodal backdoor defense methods that typically require the availability of poisoned data or involve fine-tuning the entire model, RVPT leverages few-shot downstream clean samples and only tunes a small number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\% of the parameters in CLIP, yet it significantly outperforms state-of-the-art defense methods, reducing the attack success rate from 89.70\% to 2.76\% against the most advanced multimodal attacks on ImageNet and effectively generalizes its defensive capabilities across multiple datasets.
Paper Structure (29 sections, 5 equations, 9 figures, 12 tables)

This paper contains 29 sections, 5 equations, 9 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Approach
  4. Basic Settings
  5. Clarification and practical motivation.
  6. Backdoor Attacks on CLIP
  7. Repulsive Visual Prompt Tuning
  8. Experiments
  9. Experiment Settings
  10. The Effectiveness of RVPT
  11. The Generalization of RVPT's Defensive Ability
  12. Ablation Studies
  13. Further Explorations
  14. Conclusion
  15. Why Using VPT to defend backdoored CLIP?
  16. ...and 14 more sections

Figures (9)

  • Figure 1: Perturbation Resistivity (PR) across different layers of the encoders under various perturbations, including the trigger pattern. A higher PR value indicates less sensitivity to input perturbation. Four encoder variants are evaluated on ImageNet: (a) ViT in backdoored CLIP radford2021clip. (b) ViT in backdoored CLIP tuned with Visual Prompt Tuning (VPT) on ImageNet. (c) ViT of the backdoored CLIP tuned with our RVPT on ImageNet. (d) ViT trained exclusively on clean ImageNet paszke2019pytorch. Specifically, CLIP is backdoored by the trigger of BadCLIP liang2024badclip. Detailed experimental settings and additional PR results for CLIP backdoored by other triggers can be found in \ref{['app:pr']}.
  • Figure 2: Illustration of RVPT. RVPT concatenates the features from later layers with tunable visual prompts while keeping the original parameters of CLIP frozen. To optimize the visual prompts, RVPT employs both cross-entropy (CE) loss and feature-repelling (FR) loss. The FR loss minimizes the mean cosine similarity between the prompted features and the original features across layers. Meanwhile, the CE loss ensures the clean accuracy of the prompted model. Together, these losses guide the model to encode only in-dataset predictive features that contribute to CE loss optimization, thereby enhancing its backdoor robustness.
  • Figure 3: ASR and CA are evaluated when the hyperparameters of shots, $\alpha$, context token length, depth, and margin are changed in RVPT. In panel (f), the dashed lines represent the situation when the total FR loss is ablated. More ablation studies of visual encoder architecture and handcrafted text prompt can be referred to in \ref{['app:abl']}.
  • Figure 4: Last-layer attention map for (a) original (b) poisoned image in backdoored model (attacked by BadCLIP) and RVPT.
  • Figure 5: The t-SNE JMLR:v9:vandermaaten08a plots for the representations of clean (dotted) and poisoned (star-shaped) images (attacked by BadCLIP).
  • ...and 4 more figures