When and Where to Attack? Stage-wise Attention-Guided Adversarial Attack on Large Vision Language Models

TL;DR

This work tackles the vulnerability of Large Vision-Language Models to adversarial perturbations by exploiting cross-modal attention signals. It reveals that regions with higher attention are more sensitive to adversarial changes and that attacking these hotspots induces a structured redistribution of attention toward subsequent salient areas. The authors propose Stage-wise Attention-Guided Attack (SAGA), which precomputes an attention map from an open-source LVLM and executes a stage-wise, hotspot-focused perturbation strategy under an $L_ty$ budget to maximize target-aligned outputs. Across ten LVLM targets, SAGA achieves state-of-the-art attack success and superior imperceptibility, with transferable insights suggesting attention patterns as a model-agnostic vulnerability signal. The work highlights potential defense directions and the role of attention signals in understanding and mitigating multimodal safety risks.

Abstract

Adversarial attacks against Large Vision-Language Models (LVLMs) are crucial for exposing safety vulnerabilities in modern multimodal systems. Recent attacks based on input transformations, such as random cropping, suggest that spatially localized perturbations can be more effective than global image manipulation. However, randomly cropping the entire image is inherently stochastic and fails to use the limited per-pixel perturbation budget efficiently. We make two key observations: (i) regional attention scores are positively correlated with adversarial loss sensitivity, and (ii) attacking high-attention regions induces a structured redistribution of attention toward subsequent salient regions. Based on these findings, we propose Stage-wise Attention-Guided Attack (SAGA), an attention-guided framework that progressively concentrates perturbations on high-attention regions. SAGA enables more efficient use of constrained perturbation budgets, producing highly imperceptible adversarial examples while consistently achieving state-of-the-art attack success rates across ten LVLMs. The source code is available at https://github.com/jackwaky/SAGA.

Figures (18)

• Figure 1: Example captioning responses from commercial and open-source LVLMs to images attacked using our method.
• Figure 2: All LVLMs show a statistically significant positive correlation between attention score and adversarial loss change, indicating larger loss changes for high-attention regions.
• Figure 3: Attention redistribution under different attack strategies. (a) Attention map of the image, where the red and orange boxes indicate the top 10% and top 20% attention regions, respectively. Under random cropping, attention decreases in both regions. In contrast, attacking high-attention 10% regions leads to increased attention in the next high-attention regions (20%).
• Figure 4: Overview of SAGA. Given a source image, we first extract an attention map using an open-source LVLM and pre-compute stage-wise attention hotspots before the attack. At each stage, we crop the image with the current hotspot and optimize the adversarial perturbation within the cropped region. We progressively expand the attacked region across stages, guiding the optimization toward newly emerging high-attention regions.
• Figure 5: Hotspot vs. coldspot comparison on Qwen3-VL-235B-A22B-Instruct and Gemini-2.5-Flash models.