Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models

Peng-Fei Zhang, Zi Huang

TL;DR

This paper tackles the robustness gap of Vision-Language Models by proposing Hierarchical Refinement Attack (HRA), a universal multimodal adversarial framework that refines perturbations at both the sample and optimization levels. The method disentangles image data, introduces ScMix for semantic-preserving diversity, reinforces both global and local perturbation utilities, and uses a future-aware momentum to regularize the optimization path; it also includes a text attack based on intra- and inter-sentence word importance. Extensive experiments across multiple VLP models, tasks, and datasets demonstrate that HRA achieves superior transferability compared to strong baselines, with ablation studies confirming the contribution of each component. The work reveals important insights into cross-modal vulnerabilities and provides a practical approach to evaluating and potentially mitigating such attacks, while also acknowledging limitations in text perturbation imperceptibility and budget-constrained transferability.

Abstract

Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. HRA refines universal adversarial perturbations (UAPs) at both the sample level and the optimization level. For the image modality, we disentangle adversarial examples into clean images and perturbations, allowing each component to be handled independently for more effective disruption of cross-modal alignment. We further introduce a ScMix augmentation strategy that diversifies visual contexts and strengthens both global and local utility of UAPs, thereby reducing reliance on spurious features. In addition, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, HRA identifies globally influential words by combining intra-sentence and inter-sentence importance measures, and subsequently utilizes these words as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets demonstrate the superiority of the proposed universal multimodal attacks.

Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models

TL;DR

This paper tackles the robustness gap of Vision-Language Models by proposing Hierarchical Refinement Attack (HRA), a universal multimodal adversarial framework that refines perturbations at both the sample and optimization levels. The method disentangles image data, introduces ScMix for semantic-preserving diversity, reinforces both global and local perturbation utilities, and uses a future-aware momentum to regularize the optimization path; it also includes a text attack based on intra- and inter-sentence word importance. Extensive experiments across multiple VLP models, tasks, and datasets demonstrate that HRA achieves superior transferability compared to strong baselines, with ablation studies confirming the contribution of each component. The work reveals important insights into cross-modal vulnerabilities and provides a practical approach to evaluating and potentially mitigating such attacks, while also acknowledging limitations in text perturbation imperceptibility and budget-constrained transferability.

Abstract

Existing adversarial attacks for VLP models are mostly sample-specific, resulting in substantial computational overhead when scaled to large datasets or new scenarios. To overcome this limitation, we propose Hierarchical Refinement Attack (HRA), a multimodal universal attack framework for VLP models. HRA refines universal adversarial perturbations (UAPs) at both the sample level and the optimization level. For the image modality, we disentangle adversarial examples into clean images and perturbations, allowing each component to be handled independently for more effective disruption of cross-modal alignment. We further introduce a ScMix augmentation strategy that diversifies visual contexts and strengthens both global and local utility of UAPs, thereby reducing reliance on spurious features. In addition, we refine the optimization path by leveraging a temporal hierarchy of historical and estimated future gradients to avoid local minima and stabilize universal perturbation learning. For the text modality, HRA identifies globally influential words by combining intra-sentence and inter-sentence importance measures, and subsequently utilizes these words as universal text perturbations. Extensive experiments across various downstream tasks, VLP models, and datasets demonstrate the superiority of the proposed universal multimodal attacks.
Paper Structure (37 sections, 9 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 37 sections, 9 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Vision-Language Pre-training
  4. Adversarial Attack
  5. white-box and black-box attacks
  6. Unimodal and multi-modal attacks
  7. Adversarial attack against VLP models
  8. PROPOSED METHOD
  9. Preliminaries
  10. Vanilla Learning
  11. Overview
  12. Disentangled Adversarial Images
  13. ScMix Augmentation.
  14. Local Utility Enhancement.
  15. Overall objective.
  16. ...and 22 more sections

Figures (7)

  • Figure 1: An illustration of the proposed HRA framework, which hierarchically refines universal perturbations at both the sample level and the optimization level to improve transferability. Adversarial images consisting of the original images and adversarial perturbations are optimized in a disentangled manner to enlarge their discrepancy from the original paired data. The original images are augmented using ScMix, while the local utility of adversarial perturbations is enhanced. In addition, future-aware momentum leverages both historical and predicted future gradients to regularize the current gradient when refining universal image perturbations. Meanwhile, universal text perturbations are learned by intra- and inter-sentence importance measures and ranking.
  • Figure 2: The attack success rate in terms of the average of R@1 in image-text retrieval on Flickr30K under different magnitudes of the image UAPs. The source model is ViT-B/16-based CLIP.
  • Figure 3: The attack success rate in terms of the average of R@1 in image-text retrieval on Flickr30K under different magnitudes of the text UAPs. The source model is ViT-B/16-based CLIP.
  • Figure 4: The attack success rate in terms of the average of R@1 in image-text retrieval on Flickr30K under different weights of the past and future gradients. The source model is ViT-B/16-based CLIP.
  • Figure 5: The attack success rate in terms of the average of R@1 in image-text retrieval on Flickr30K under different numbers of future gradient steps. The source model is ViT-B/16-based CLIP.
  • ...and 2 more figures