Attention-Guided Patch-Wise Sparse Adversarial Attacks on Vision-Language-Action Models

TL;DR

The paper introduces ADVLA, a gray-box attack framework that perturbs visual-encoder projections into the textual space of Vision-Language-Action models. By integrating three attention-guided strategies—Attention-Weighted Gradient, Top-K Masked Gradient, and Top-K Loss—ADVLA achieves high attack success under low-amplitude constraints while maintaining sparsity and perceptual stealth. Experiments on the LIBERO benchmark with OpenVLA show nearly complete disruption of downstream actions at $L_1 \,=\ 4/255$ within a few inner-loop iterations, with perturbations concentrated on critical regions and runtimes orders of magnitude faster than prior patch-based methods. The work emphasizes the need for robust defenses in embodied AI and demonstrates strong practical value for evaluating VLA robustness in feature space.

Abstract

In recent years, Vision-Language-Action (VLA) models in embodied intelligence have developed rapidly. However, existing adversarial attack methods require costly end-to-end training and often generate noticeable perturbation patches. To address these limitations, we propose ADVLA, a framework that directly applies adversarial perturbations on features projected from the visual encoder into the textual feature space. ADVLA efficiently disrupts downstream action predictions under low-amplitude constraints, and attention guidance allows the perturbations to be both focused and sparse. We introduce three strategies that enhance sensitivity, enforce sparsity, and concentrate perturbations. Experiments demonstrate that under an $L_{\infty}=4/255$ constraint, ADVLA combined with Top-K masking modifies less than 10% of the patches while achieving an attack success rate of nearly 100%. The perturbations are concentrated on critical regions, remain almost imperceptible in the overall image, and a single-step iteration takes only about 0.06 seconds, significantly outperforming conventional patch-based attacks. In summary, ADVLA effectively weakens downstream action predictions of VLA models under low-amplitude and locally sparse conditions, avoiding the high training costs and conspicuous perturbations of traditional patch attacks, and demonstrates unique effectiveness and practical value for attacking VLA feature spaces.

Figures (2)

• Figure 1: The pipeline of ADVLA. In Module (a), it is the main attack method of ADVLA, first, in every VLA step, we got original image from the environment and initialize random noise on it to create initial adversaial image, then we put original image in vision backbone and projector to generate clean vision feature, next, we iterate to put adversarial image in vision encoder to generate adversarial feature, and calculate similarity loss, so we can iterate perturbation based on Projection Gradient Descent (PGD) method and update adversarial image. In (b), we generate attention weight mask from vision backbone, which is composed of two Vision Transformers (ViT): DinoV2 oquab2023dinov2 and SigLIPzhai2023sigmoid, openvla fused them to generate vision embeddings and project concated embddings to text feature space, we use one of the two ViTs to hook attention map and resize it to the size of image, then we set patchs which have top-k weights to 1 and others to 0, so we got attention mask, and then flatten it to get loss mask. (c), we show three methods, which use three mask on noise update and loss calculation respectively.
• Figure 2: Image samples in experiments. For visualization purposes, all perturbation images are amplified for clarity (for display only). The column (a): UADAwang2024exploring, which apply an unlimited trained patch on observation images, and it's very easy to be found. The column (b): ADVLA with fully noise, it's easy to see there is noise on it. The column (c): ADVLA-TKM, we select top-10% patches based on attention weight, as we see, the noise is almost invisible, just noise on top-k patches. Last column is the visualized attention weight of vision-backbone in vla, as shown in pictures, ViTs always mainly focus on the robot arms.