Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack

TL;DR

The paper addresses the limited transferability of multimodal adversarial examples in Vision-Language Pre-training (VLP) models by identifying input-diversity overfitting as a key issue. It introduces Local Shuffle and Sample-based Attack (LSSA), which expands image-text input pairs through local block shuffles and neighborhood sampling to craft more transferable adversarial perturbations, with both image and text components optimized jointly. Empirical results on Flickr30K and MSCOCO across fused and aligned VLPs, as well as LVLMs, show that LSSA significantly outperforms prior attacks in white-box and black-box settings and exhibits strong cross-task transferability. The work provides a robust framework for evaluating adversarial robustness in VLPs and motivates further defense-focused research for multimodal systems.

Abstract

Visual-Language Pre-training (VLP) models have achieved significant performance across various downstream tasks. However, they remain vulnerable to adversarial examples. While prior efforts focus on improving the adversarial transferability of multimodal adversarial examples through cross-modal interactions, these approaches suffer from overfitting issues, due to a lack of input diversity by relying excessively on information from adversarial examples in one modality when crafting attacks in another. To address this issue, we draw inspiration from strategies in some adversarial training methods and propose a novel attack called Local Shuffle and Sample-based Attack (LSSA). LSSA randomly shuffles one of the local image blocks, thus expanding the original image-text pairs, generating adversarial images, and sampling around them. Then, it utilizes both the original and sampled images to generate the adversarial texts. Extensive experiments on multiple models and datasets demonstrate that LSSA significantly enhances the transferability of multimodal adversarial examples across diverse VLP models and downstream tasks. Moreover, LSSA outperforms other advanced attacks on Large Vision-Language Models.

Figures (10)

• Figure 1: Comparison of attack success rate (%) using our LSSA method and existing advanced attacks in image-text retrieval tasks. The multimodal adversarial examples are crafted on the ALBEF model to attack ALBEF, TCL, CLIPViT and CLIPCNN, respectively.
• Figure 2: Comparison of SGA and LSSA. (a) and (b) show the multimodal adversarial examples generation process of SGA and LSSA, respectively. V' and T' represent the corresponding image and text adversarial examples. Dashed lines enclose the expanded set. Arrows indicate the process or guidance for generating adversarial examples.
• Figure 3: The attack success rate of multimodal adversarial examples against TCL model, which are crafted on ALBEF model. We explore three methods: a) Original SGA three steps attack (t-i-t), b) SGA two steps attack (i-t): generating adversarial images using original text and then generating adversarial text using the adversarial images, c) Sample-based SGA two steps attack (i-t): generating adversarial images using original text and then using the original and neighbours of the adversarial image to generate the adversarial images.
• Figure 4: The attack success rate (%) of LSSA with different shuffle number $N$ on the Flickr30K dataset. The source model is ALBEF, and the target model is other VLP models.
• Figure 5: The attack success rate (%) of LSSA with different shuffle position on the Flickr30K dataset. The source model is ALBEF, and the target model is other VLP models.