VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models

TL;DR

VLAttack tackles the practical problem of attacking black-box downstream tasks by only exploiting publicly available vision-language pre-trained models. It introduces a two-stage attack: a single-modal phase with block-wise similarity attacks on images (BSA) and text perturbations via BERT-Attack, followed by an iterative multimodal cross-search attack (ICSA) that jointly refines image-text perturbations under remaining budget. Across five VL models and six tasks, VLAttack achieves the highest attack success rates, significantly outperforming baselines and exposing robustness blind spots in current VL systems. The work highlights the need for robust transferability-aware defenses in vision-language models and provides public code for reproducibility.

Abstract

Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels. At the single-modal level, we propose a new block-wise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models. Source codes can be found at https://github.com/ericyinyzy/VLAttack.

Figures (20)

• Figure 1: An illustration of the problem of attacking block-box downstream tasks using pre-trained vision-language models.
• Figure 2: A brief illustration of VLAttack.
• Figure 3: A brief illustration of the encoder-only (a) and encoder-decoder (b) structures.
• Figure 4: Block-wise similarity attack. $\mathbf{F}_{\alpha}$ is the image encoder, and $\mathbf{F}_{\beta}$ is the Transformer encoder.
• Figure 5: Ablation analysis of different components in VLAttack. We show the results of VQAv2 (a) and SNLI-VE (b) on OFA, and VQAv2 (c) and RefCOCO (d) on Unitab.