Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models

Peng-Fei Zhang, Guangdong Bai, Zi Huang

TL;DR

Vision-language pre-trained models face robustness challenges due to poor adversarial transferability across architectures. MAAs approach—combining RScrop for fine-grained image exploration with MGSD to widen embedding distances across layers and modalities—yields highly transferable attacks, complemented by BERT-Attack for text. Across Flickr30K, MSCOCO, and RefCOCO+, MAA outperforms baselines on image-text retrieval, visual grounding, and image captioning, while ablations confirm the complementary value of RScrop and MGSD. This lightweight, sample-driven framework strengthens robustness evaluation of multi-modal systems and informs design choices for more resilient vision-language models.

Abstract

Current adversarial attacks for evaluating the robustness of vision-language pre-trained (VLP) models in multi-modal tasks suffer from limited transferability, where attacks crafted for a specific model often struggle to generalize effectively across different models, limiting their utility in assessing robustness more broadly. This is mainly attributed to the over-reliance on model-specific features and regions, particularly in the image modality. In this paper, we propose an elegant yet highly effective method termed Meticulous Adversarial Attack (MAA) to fully exploit model-independent characteristics and vulnerabilities of individual samples, achieving enhanced generalizability and reduced model dependence. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique, incorporating a multi-granularity similarity disruption (MGSD) strategy. Extensive experiments across diverse VLP models, multiple benchmark datasets, and a variety of downstream tasks demonstrate that MAA significantly enhances the effectiveness and transferability of adversarial attacks. A large cohort of performance studies is conducted to generate insights into the effectiveness of various model configurations, guiding future advancements in this domain.

MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models

TL;DR

Vision-language pre-trained models face robustness challenges due to poor adversarial transferability across architectures. MAAs approach—combining RScrop for fine-grained image exploration with MGSD to widen embedding distances across layers and modalities—yields highly transferable attacks, complemented by BERT-Attack for text. Across Flickr30K, MSCOCO, and RefCOCO+, MAA outperforms baselines on image-text retrieval, visual grounding, and image captioning, while ablations confirm the complementary value of RScrop and MGSD. This lightweight, sample-driven framework strengthens robustness evaluation of multi-modal systems and informs design choices for more resilient vision-language models.

Abstract

Current adversarial attacks for evaluating the robustness of vision-language pre-trained (VLP) models in multi-modal tasks suffer from limited transferability, where attacks crafted for a specific model often struggle to generalize effectively across different models, limiting their utility in assessing robustness more broadly. This is mainly attributed to the over-reliance on model-specific features and regions, particularly in the image modality. In this paper, we propose an elegant yet highly effective method termed Meticulous Adversarial Attack (MAA) to fully exploit model-independent characteristics and vulnerabilities of individual samples, achieving enhanced generalizability and reduced model dependence. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique, incorporating a multi-granularity similarity disruption (MGSD) strategy. Extensive experiments across diverse VLP models, multiple benchmark datasets, and a variety of downstream tasks demonstrate that MAA significantly enhances the effectiveness and transferability of adversarial attacks. A large cohort of performance studies is conducted to generate insights into the effectiveness of various model configurations, guiding future advancements in this domain.

Paper Structure

This paper contains 19 sections, 4 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Vision-Language Pre-trained Models
  4. Adversarial Attack
  5. Methodology
  6. Preliminaries
  7. Meticulous adversarial attack
  8. Experiments
  9. Settings
  10. Task analysis
  11. Results on the Image-Text Retrieval
  12. Results on the Visual Grounding and Image Captioning
  13. Parameter Analysis
  14. Effect of Varying Perturbation Magnitudes
  15. Effect of Resizing Factors
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An illustration of the proposed (a) RScrop augmentation and (b) MAA method. For RScrop, we use the ViT-based model as an example, where images are processed patch-wise. RScrop zooms in on adversarial examples and applies them in a sliding manner along each dimension (with the x-dimension as an example) to capture more fine-grained local regions and their interconnections. The crop window shifts by a small step size and moves to adjacent non-overlapping areas relative to its previous position, repeating this process until the entire image is covered. MAA enlarges the feature distance between adversarial images and the original images across various layers and components of the model, while also maximizing the cross-modal gap.
  • Figure 2: Test accuracy on Flickr30K with varying image perturbation magnitudes. The source model is CLIP$_\text{ViT-B/16}$. The attack success rate (ASR, $\%$) of R@1 is reported.
  • Figure 3: (a) Resizing parameter analysis: appropriate scaling can achieve competitive performance. (b) A Grad-CAM visualization of original data pairs, image-only perturbed pairs and multi-modal perturbed pairs, where MAA can significantly shift the attention of target models.