Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

TL;DR

This work investigates adversarial transferability in vision-language pre-training (VLP) models through the lens of modality interaction. It introduces the Collaborative Multimodal Interaction Attack (CMI-Attack), which combines Embedding Guidance and Interaction Enhancement to exploit cross-modal correlations, notably perturbing text in embedding space and leveraging image gradients to constrain multimodal perturbations. The method achieves superior transferability across diverse VLP architectures on Flickr30K and MSCOCO, including notable gains in both image-text retrieval and cross-task image captioning, and is supported by comprehensive ablations and visualizations showing imperceptible perturbations. Overall, the study highlights modality interaction as a key factor in adversarial effectiveness and calls for robust defenses that address cross-modal dynamics in VLP models.

Abstract

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, $\text{CLIP}_{\text{ViT}}$ and $\text{CLIP}_{\text{CNN}}$ by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness.

Figures (5)

• Figure 1: Comparison of Attack Schemes for Different Attack. Each subplot presents the attack process of the corresponding method on both images and text under specific conditions. Our approach enhances attack performance by fully leveraging the information generated through the multimodal interaction process.
• Figure 2: Comparing the attack success rate (ASR) on the IR R@1 metric results for image-text retrieval using different text attacks. The two rows of bar plots display the transfer attack success rates for Co-attack and SGA, respectively.
• Figure 3: Comparing the impact of Interaction Gradient Information (IGI) on the attack success rates for the IR R@1 metric in image-text retrieval. The first column of each bar chart represents the original method, while the second column represents the original method with interaction gradient information.
• Figure 4: Impact of Iteration Count $N$ on Interactive Enhancement. We use the Flickr30K dataset, utilizing ALBEF as the source model and having both ALBEF and TCL as distinct target models. The trend indicates an overall improvement in ASR (%) with the increasing $N$.
• Figure 5: Visualization of CMI-Attack. The first row presents the original images, the second row displays adversarial examples generated by CMI-Attack, and the third row shows perturbations magnified 50 times for better visualization. The last two rows include the original caption and the caption after perturbation, respectively.