Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Prompt Tuning for Vision-Language Models

Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, Jitao Sang

TL;DR

This paper tackles the vulnerability of vision–language models to adversarial image perturbations by proposing Adversarial Prompt Tuning (AdvPT). AdvPT uses learnable text prompts to align clean text embeddings with adversarial image embeddings, constructing an adversarial embedding bank from a frozen image encoder and optimizing the prompts through the text encoder to improve robustness without architectural changes. It demonstrates strong white-box and black-box robustness across eight datasets, and shows compatibility and additive benefits when combined with image-based defenses like DiffPure, while highlighting a generalization–robustness trade-off and domain-transfer properties. The approach is computationally efficient, requiring training only the prompt vectors and not the image encoder, and offers practical implications for secure, scalable multimodal systems, with further insights into the linguistic and geometric nature of the learned prompts. Overall, AdvPT advances robustness in VLMs by exploiting textual input modifications, opening new directions for defense strategies that complement traditional image-space defenses and prompt-learning methods, with potential extensions to broader multimodal tasks.

Abstract

With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities. However, these models remain vulnerable to adversarial attacks, particularly in the image modality, presenting considerable security risks. This paper introduces Adversarial Prompt Tuning (AdvPT), a novel technique to enhance the adversarial robustness of image encoders in VLMs. AdvPT innovatively leverages learnable text prompts and aligns them with adversarial image embeddings, to address the vulnerabilities inherent in VLMs without the need for extensive parameter training or modification of the model architecture. We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques, further boosting defensive capabilities. Comprehensive experimental analyses provide insights into adversarial prompt tuning, a novel paradigm devoted to improving resistance to adversarial images through textual input modifications, paving the way for future robust multimodal learning research. These findings open up new possibilities for enhancing the security of VLMs. Our code is available at https://github.com/jiamingzhang94/Adversarial-Prompt-Tuning.

Adversarial Prompt Tuning for Vision-Language Models

TL;DR

This paper tackles the vulnerability of vision–language models to adversarial image perturbations by proposing Adversarial Prompt Tuning (AdvPT). AdvPT uses learnable text prompts to align clean text embeddings with adversarial image embeddings, constructing an adversarial embedding bank from a frozen image encoder and optimizing the prompts through the text encoder to improve robustness without architectural changes. It demonstrates strong white-box and black-box robustness across eight datasets, and shows compatibility and additive benefits when combined with image-based defenses like DiffPure, while highlighting a generalization–robustness trade-off and domain-transfer properties. The approach is computationally efficient, requiring training only the prompt vectors and not the image encoder, and offers practical implications for secure, scalable multimodal systems, with further insights into the linguistic and geometric nature of the learned prompts. Overall, AdvPT advances robustness in VLMs by exploiting textual input modifications, opening new directions for defense strategies that complement traditional image-space defenses and prompt-learning methods, with potential extensions to broader multimodal tasks.

Abstract

With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities. However, these models remain vulnerable to adversarial attacks, particularly in the image modality, presenting considerable security risks. This paper introduces Adversarial Prompt Tuning (AdvPT), a novel technique to enhance the adversarial robustness of image encoders in VLMs. AdvPT innovatively leverages learnable text prompts and aligns them with adversarial image embeddings, to address the vulnerabilities inherent in VLMs without the need for extensive parameter training or modification of the model architecture. We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques, further boosting defensive capabilities. Comprehensive experimental analyses provide insights into adversarial prompt tuning, a novel paradigm devoted to improving resistance to adversarial images through textual input modifications, paving the way for future robust multimodal learning research. These findings open up new possibilities for enhancing the security of VLMs. Our code is available at https://github.com/jiamingzhang94/Adversarial-Prompt-Tuning.
Paper Structure (30 sections, 3 equations, 7 figures, 5 tables, 1 algorithm)

This paper contains 30 sections, 3 equations, 7 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Language Models
  4. Prompt Learning
  5. Adversarial Defenses
  6. Revisiting Clip and the Adversarial Robustness of Its Image Encoder
  7. CLIP
  8. Adversarial Robustness of CLIP’s Image Encoder
  9. Adversarial Prompt Tuning
  10. Overview.
  11. Adversarial Embedding Bank Generation
  12. Learnable Vector Optimization
  13. Experiments
  14. Experimental Settings
  15. Datasets.
  16. ...and 15 more sections

Figures (7)

  • Figure 1: The defending effect of AdvPT: hand-crafted prompts (top) fail to match with adversarial images, whereas prompts constituted by learnable vectors (bottom) enable correct recognition.
  • Figure 2: An overview of the AdvPT framework.
  • Figure 3: AdvPT vs. CoOp on generalization and adversarial robustness.
  • Figure 4: Efficiency comparison between AdvPT and AT on Pets.
  • Figure 5: Efficiency comparison between AdvPT and Fast AT on Pets.
  • ...and 2 more figures