Goal-oriented Backdoor Attack against Vision-Language-Action Models via Physical Objects

TL;DR

This work exposes a practical backdoor threat to vision-language-action (VLA) models by poisoning training data with physical object triggers, enabling goal-oriented actions when triggers appear while preserving normal behavior otherwise. It introduces GoBA and the BadLIBERO dataset built on LIBERO, plus a three-level evaluation to capture progress from no action to successful backdoor execution. Empirically, GoBA achieves near-perfect level-3 backdoor success across evaluated VLAs and tasks, with minimal impact on clean inputs, and reveals how action trajectories, trigger color, and object choice modulate attack strength. The results stress the need for robust data curation and potential defenses (e.g., filtering by end-position trajectories) to secure embodied AI systems in real-world deployments.

Abstract

Recent advances in vision-language-action (VLA) models have greatly improved embodied AI, enabling robots to follow natural language instructions and perform diverse tasks. However, their reliance on uncurated training datasets raises serious security concerns. Existing backdoor attacks on VLAs mostly assume white-box access and result in task failures instead of enforcing specific actions. In this work, we reveal a more practical threat: attackers can manipulate VLAs by simply injecting physical objects as triggers into the training dataset. We propose goal-oriented backdoor attacks (GoBA), where the VLA behaves normally in the absence of physical triggers but executes predefined and goal-oriented actions in the presence of physical triggers. Specifically, based on a popular VLA benchmark LIBERO, we introduce BadLIBERO that incorporates diverse physical triggers and goal-oriented backdoor actions. In addition, we propose a three-level evaluation that categorizes the victim VLA's actions under GoBA into three states: nothing to do, try to do, and success to do. Experiments show that GoBA enables the victim VLA to successfully achieve the backdoor goal in 97 percentage of inputs when the physical trigger is present, while causing zero performance degradation on clean inputs. Finally, by investigating factors related to GoBA, we find that the action trajectory and trigger color significantly influence attack performance, while trigger size has surprisingly little effect. The code and BadLIBERO dataset are accessible via the project page at https://goba-attack.github.io/.

Figures (13)

• Figure 1: Comparison between prior backdoor attacks and our proposed method. All demonstrations under the same instruction: "Pick up the alphabet soup and place it in the basket." (b) BadVLA zhou2025badvla employs a patch-based trigger (highlighted with a red box), which leads to random actions. (c) Our attack instead utilizes a physical object as the trigger (highlighted with a red box) and enforces a goal-oriented behavior, such as picking up the trigger object (cookie) and placing it on the right side of the operating surface.
• Figure 2: One of the tasks with three different backdoor action trajectories. For all three backdoor demostration of this task, the language instruction remains unchanged: "Pick up the alphabet soup and place it in the basket."
• Figure 3: Color tests. The backdoor action trajectory is fixed to trajectory $1$ (see Figure \ref{['fig:action_test']}).
• Figure 4: Size test. To eliminate potential bias introduced by the varying difficulty of grasping triggers of different sizes, we fix the action trajectory to pick up the target object and place it in the predefined region (see Figure \ref{['fig:action_test']}, action trajectory $3$).
• Figure 5: Cross-evaluation of different trigger packaging. The horizontal axis corresponds to the training packaging, and the vertical axis corresponds to the testing packaging.