A Two-Stage Globally-Diverse Adversarial Attack for Vision-Language Pre-training Models

TL;DR

This work tackles the vulnerability of vision-language pre-training models to adversarial perturbations, especially in black-box settings, by introducing 2S-GDA, a two-stage globally-diverse attack. It replaces unstable textual re-perturbation in prior three-stage schemes with Stage I text perturbations guided by a globally-diverse mechanism and Stage II robust visual perturbations via multi-scale resizing and block-shuffle rotation. The Globally-Diverse Strategy combines Candidate Text Expansion and Globally-Aware Replacement to explore a richer semantic perturbation space, enhancing cross-modal misalignment captured by a cross-modal alignment objective. Extensive experiments across multiple VLP architectures show that 2S-GDA improves transferability over state-of-the-art baselines, with up to 11.17 percentage-point gains in black-box settings, and demonstrates modularity by further boosting attack success when integrated with existing methods. The approach offers a practical, extensible way to evaluate and potentially reinforce the robustness of multimodal retrieval and alignment in VLP systems.

Abstract

Vision-language pre-training (VLP) models are vulnerable to adversarial examples, particularly in black-box scenarios. Existing multimodal attacks often suffer from limited perturbation diversity and unstable multi-stage pipelines. To address these challenges, we propose 2S-GDA, a two-stage globally-diverse attack framework. The proposed method first introduces textual perturbations through a globally-diverse strategy by combining candidate text expansion with globally-aware replacement. To enhance visual diversity, image-level perturbations are generated using multi-scale resizing and block-shuffle rotation. Extensive experiments on VLP models demonstrate that 2S-GDA consistently improves attack success rates over state-of-the-art methods, with gains of up to 11.17\% in black-box settings. Our framework is modular and can be easily combined with existing methods to further enhance adversarial transferability.

Figures (5)

• Figure 1: Comparison of cross-modal interactions. (a) SGA: three-stage perturbation (S1: text $\rightarrow$ S2: image $\rightarrow$ S3: text). (b) 2S-GDA: two-stage perturbation (S1: text $\rightarrow$ S2: image). $v$: input image; $t$: paired caption; $t^{*}$: intermediate state; $v'$, $t'$: corresponding adversarial examples. Arrows indicate guidance for generating adversarial examples.
• Figure 2: Feature-space analysis of SGA.
• Figure 3: Visualization of adversarial examples on the image-text retrieval task (ALBEF→CLIPViT).
• Figure 4: Visualization of clean vs. adversarial examples. Rows show captions, images, and salient regions for “boy”.
• Figure 5: Comparison of attack success rates under different number of influential tokens $k$ in GAR, with the baseline (w/o GAR) included for reference (ALBEF→CLIPViT).