An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models

TL;DR

This work studies cross-prompt adversarial transferability in vision-language models (VLMs)—the ability of a single adversarial image to mislead predictions across a thousand textual prompts. It introduces Cross-Prompt Attack (CroPA), a min-max framework that jointly optimizes a visual perturbation and a learnable prompt perturbation to maximize attack success across prompts while keeping the image perturbation within imperceptible bounds. CroPA significantly improves cross-prompt transferability over baseline single- and multi-prompt attacks and is validated across Flamingo, BLIP-2, and InstructBLIP on classification, captioning, and VQA tasks; analyses reveal that CroPA broadens the prompt-embedding coverage beyond human-interpretable prompts. These insights highlight potential security risks in VLMs and point to future defenses and optimization strategies, including query-based attack approaches.”

Abstract

Different from traditional task-specific vision models, recent large VLMs can readily adapt to different vision tasks by simply using different textual instructions, i.e., prompts. However, a well-known concern about traditional task-specific vision models is that they can be misled by imperceptible adversarial perturbations. Furthermore, the concern is exacerbated by the phenomenon that the same adversarial perturbations can fool different task-specific models. Given that VLMs rely on prompts to adapt to different tasks, an intriguing question emerges: Can a single adversarial image mislead all predictions of VLMs when a thousand different prompts are given? This question essentially introduces a novel perspective on adversarial transferability: cross-prompt adversarial transferability. In this work, we propose the Cross-Prompt Attack (CroPA). This proposed method updates the visual adversarial perturbation with learnable prompts, which are designed to counteract the misleading effects of the adversarial image. By doing this, CroPA significantly improves the transferability of adversarial examples across prompts. Extensive experiments are conducted to verify the strong cross-prompt adversarial transferability of CroPA with prevalent VLMs including Flamingo, BLIP-2, and InstructBLIP in various different tasks. Our source code is available at \url{https://github.com/Haochen-Luo/CroPA}.

Figures (5)

• Figure 1: Overview of CroPA’s framework under the targeted attack setting. Both the image perturbation $\delta_v$ and the prompt perturbation $\delta_t$ are learnable but the prompt perturbation does not collaborate with $\delta_v$ to deceive the model. They are optimised with the opposite goals: $\delta_v$ aims to minimise the language modelling loss while the $\delta_t$ aims to maximise the language modelling loss. The update frequency of the image perturbation and prompt perturbation can be different.
• Figure 2: The targeted ASR of three methods tested on (a) Flamingo, (b) BLIP-2, and (c) InstructBLIP. Different numbers of prompts 1, 5, 10, 50, and 100 are used in the transferability test. Our CroPA achieve better cross-prompt adversarial transferability than Single-P and Multi-P.
• Figure 3: Targeted ASRs of Baseline Methods (Single-P and Multi-P) and CroPA over attack Iterations. With the same number of attack iterations, our CroPA significantly outperforms baselines.
• Figure 4: Visualisation of the prompt embeddings with t-SNE van2008visualizing.
• Figure 5: Effect of different update step size of prompt perturbation of CroPAjoint