Model-agnostic Adversarial Attack and Defense for Vision-Language-Action Models

TL;DR

This work reveals critical vulnerabilities of Vision-Language-Action (VLA) models to patch-based adversarial attacks and introduces Embedding Disruption Patch Attack (EDPA), a model-agnostic method that attacks latent embeddings to misalign visual inputs with linguistic instructions. It couples EDPA with an adversarial finetuning strategy for the visual encoder to restore robust latent alignment and preserve clean-input performance. Evaluations on LIBERO show that EDPA dramatically degrades OpenVLA-family models, while the proposed encoder finetuning substantially mitigates the damage and improves cross-attack robustness with only modest trade-offs on clean performance. The study also analyzes patch transferability across datasets and models and provides patch visualizations and attention insights, highlighting practical security implications for real-world embodied AI systems.

Abstract

Vision-Language-Action (VLA) models have achieved revolutionary progress in robot learning, enabling robots to execute complex physical robot tasks from natural language instructions. Despite this progress, their adversarial robustness remains underexplored. In this work, we propose both adversarial patch attack and corresponding defense strategies for VLA models. We first introduce the Embedding Disruption Patch Attack (EDPA), a model-agnostic adversarial attack that generates patches directly placeable within the camera's view. In comparison to prior methods, EDPA can be readily applied to different VLA models without requiring prior knowledge of the model architecture, or the controlled robotic manipulator. EDPA constructs these patches by (i) disrupting the semantic alignment between visual and textual latent representations, and (ii) maximizing the discrepancy of latent representations between adversarial and corresponding clean visual inputs. Through the optimization of these objectives, EDPA distorts the VLA's interpretation of visual information, causing the model to repeatedly generate incorrect actions and ultimately result in failure to complete the given robotic task. To counter this, we propose an adversarial fine-tuning scheme for the visual encoder, in which the encoder is optimized to produce similar latent representations for both clean and adversarially perturbed visual inputs. Extensive evaluations on the widely recognized LIBERO robotic simulation benchmark demonstrate that EDPA substantially increases the task failure rate of cutting-edge VLA models, while our proposed defense effectively mitigates this degradation. The codebase is accessible via the homepage at https://edpa-attack.github.io/.

Figures (6)

• Figure 1: Overview of OpenVLA architecture and patch attack requirements. Given a visual observation and a language instruction, the OpenVLA model first encodes the inputs into token-level latent representations. These representations are processed by the LVLM to produce action tokens, which are subsequently decoded into executable actions for the robotic platform. The colored dashed lines highlight the prior knowledge and/or access required by different patch attacks for various modules within the VLA: green for EDPA, purple for UADA, and red for UPA.
• Figure 2: Visualization of Patch. All the patches are generated on LIBERO dataset.
• Figure 3: Average attention weights of each linguistic token to the primary camera input in the first layer of OpenVLA.
• Figure 4: Average attention weights of each linguistic token to the primary camera input in the last layer of OpenVLA.
• Figure 5: Impact of patch size. The figure shows how varying patch sizes of EDPA affect the average failure rate (FR) across different VLA models on the LIBERO simulation benchmark.