Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning

TL;DR

This work addresses the security risks of visual backdoors in MLLM-based embodied agents by introducing BEAT, a framework that uses environmental object triggers to induce attacker-specified multi-step policies. BEAT constructs benign, backdoor, and contrastive data and adopts a two-stage fine-tuning process: supervised fine-tuning to acquire general competence, followed by Contrastive Trigger Learning (CTL) to sharpen trigger-boundaries and minimize false activations. Across two embodied benchmarks and multiple MLLMs, BEAT achieves attack success up to $80\%$ while preserving benign task performance, and demonstrates robust generalization to out-of-distribution trigger placements; CTL further improves backdoor activation precision by up to 39% in F1_BT and maintains high ASR with limited backdoor data. These findings reveal a critical safety vulnerability in vision-driven embodied agents and highlight the urgent need for robust defenses to ensure reliable deployment in safety-critical settings.

Abstract

Multimodal large language models (MLLMs) have advanced embodied agents by enabling direct perception, reasoning, and planning task-oriented actions from visual inputs. However, such vision driven embodied agents open a new attack surface: visual backdoor attacks, where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-specified multi-step policy. We introduce BEAT, the first framework to inject such visual backdoors into MLLM-based embodied agents using objects in the environments as triggers. Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably. BEAT addresses this challenge by (1) constructing a training set that spans diverse scenes, tasks, and trigger placements to expose agents to trigger variability, and (2) introducing a two-stage training scheme that first applies supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning (CTL). CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation. Across various embodied agent benchmarks and MLLMs, BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance, and generalizes reliably to out-of-distribution trigger placements. Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39% under limited backdoor data. These findings expose a critical yet unexplored security risk in MLLM-based embodied agents, underscoring the need for robust defenses before real-world deployment.

Figures (6)

• Figure 1: Backdoor attacks on MLLM-driven embodied agent. Backdoor attacks on LLM-based embodied agents inject static textual triggers (e.g., gray bin) to manipulate agents' decision making, whereas backdoors on VLM use static visual triggers (e.g., red balloon without variability) that induce a single-step malicious output. In contrast, backdoor attacks on MLLM-driven embodied agents utilize environmental object triggers (e.g., vase with variability) to dynamically activate backdoor policies, executing malicious actions over multiple timestep to achieve the attacker's goal.
• Figure 2: Two-stage backdoor fine-tuning scheme in BEAT. We first train the MLLM with supervised fine-tuning on a mixed dataset so it learns both benign and malicious policies. We then apply contrastive trigger learning, using a preference-paired dataset to strengthen its ability to distinguish and switch between behaviors: given the same interaction history $h$, the model prefers the benign action $a_{\mathrm{benign}}$ on trigger-free inputs ($v_{-}$) and the backdoor action $a_{\mathrm{attack}}$ on triggered inputs ($v_{+}$).
• Figure 3: Examples of successful backdoor trajectories of BEAT. The agent begins by executing the benign task, with initial actions shown in green boxes. Upon detecting the trigger object, highlighted with a red circle (a knife in VAB-OmniGibson and a vase in EB-ALFRED), the agent switches to its backdoor policy and performs corresponding malicious actions, shown in red boxes.
• Figure 4: Impact of backdoor data ratio in BEAT. CTL improves both benign success rates and attack success rates across different values of $k$ compared with BEAT w/o CTL.
• Figure 5: False triggering rate (FTR). CTL sharply reduces FTRs on benign tasks.