Boosting Transferability in Vision-Language Attacks via Diversification along the Intersection Region of Adversarial Trajectory

TL;DR

The paper tackles the vulnerability of vision-language pre-training (VLP) models to multimodal adversarial examples by extending the diversification strategy beyond the online optimization path. It introduces diversification along the intersection region of adversarial trajectories and text-guided selection, plus adversarial text that deviates from the intersection region, to reduce surrogate-model overfitting. Across Flickr30K and MSCOCO, the approach yields higher transferability of multimodal AEs to unseen VLP models and across downstream tasks like image-text retrieval, visual grounding, and image captioning, outperforming the prior SGA baseline. The findings offer a principled method to enhance attack transferability while also informing defenses and robustness research for multimodal models, with broader implications for cross-task generalization and LLM interaction.

Abstract

Vision-language pre-training (VLP) models exhibit remarkable capabilities in comprehending both images and text, yet they remain susceptible to multimodal adversarial examples (AEs). Strengthening attacks and uncovering vulnerabilities, especially common issues in VLP models (e.g., high transferable AEs), can advance reliable and practical VLP models. A recent work (i.e., Set-level guidance attack) indicates that augmenting image-text pairs to increase AE diversity along the optimization path enhances the transferability of adversarial examples significantly. However, this approach predominantly emphasizes diversity around the online adversarial examples (i.e., AEs in the optimization period), leading to the risk of overfitting the victim model and affecting the transferability. In this study, we posit that the diversity of adversarial examples towards the clean input and online AEs are both pivotal for enhancing transferability across VLP models. Consequently, we propose using diversification along the intersection region of adversarial trajectory to expand the diversity of AEs. To fully leverage the interaction between modalities, we introduce text-guided adversarial example selection during optimization. Furthermore, to further mitigate the potential overfitting, we direct the adversarial text deviating from the last intersection region along the optimization path, rather than adversarial images as in existing methods. Extensive experiments affirm the effectiveness of our method in improving transferability across various VLP models and downstream vision-and-language tasks.

Figures (9)

• Figure 1: Our method vs. set-level guided attack (SGA) lu2023setlevel. (a) shows the main idea of SGA, i.e., conducting augmentation around the online adversarial examples. (b) shows the main idea of our method, that is, we perform augmentation in the intersection region of adversarial trajectory. The red and blue dots both depict images sampled from the intersection region, with red dots indicating the best samples selected using the text-guided adversarial example selection strategy. The surrounding light red dots represent applying the same resizing data augmentation to the best samples as utilized in SGA. (c) and (d) compare the transferability of our method and SGA by using the adversarial examples of ALBEF li2021align and CLIP$_\text{ViT}$ to attack CLIP$_\text{CNN}$, respectively.
• Figure 2: Visualization on Image Captioning. We use the ALBEF model, pre-trained on Image Text Retrieval(ITR) task, to generate adversarial images on the MSCOCO dataset and use the BLIP li2022blip model for Image Captioning on both clean images and adversarial images, respectively.
• Figure 3: Visualization on Visual Grounding. We use the ALBEF model, pre-trained on the ITR task, to generate adversarial images on the RefCOCO+ dataset and use the same model, pre-trained on Visual Grouding(VG) task, to localize the regions corresponding to red words on both clean images and adversarial images, respectively.
• Figure 4: Ablation Study: Attack Success Rate(%) on other three target models. The baseline is SGA. Setting 1 removes diversification along the intersection region of adversarial trajectory. Setting 2 removes the text deviating from the last intersection region along the optimization path.
• Figure A1: Adversarial perturbation visualization on Multimodal dataset.