GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?

TL;DR

GhostEI-Bench addresses the safety gaps of mobile Vision-Language Model agents under dynamic environmental injection by introducing an executable Android-emulator benchmark with 110 test cases across 14 apps and 7 domains. It combines a unified threat model with three attack vectors (Deceptive Instruction, Static Environmental Injection, Dynamic Environmental Injection) and a judge-LM protocol to diagnose perception, recognition, and reasoning failures, yielding a Vulnerability Rate metric $\text{VR} = \frac{\text{FAS} + \text{PAS}}{\text{Total Cases} - \text{BF}}$ that isolates security from capability. The study reveals pervasive vulnerabilities across state-of-the-art models, with VR generally in the 40–55% range and GPT-5 offering the strongest overall performance and robustness, while reflection and reasoning modules provide nuanced improvements depending on the model. This benchmark provides a reproducible framework for measuring and mitigating dynamic GUI threats, guiding the development of more robust and secure embodied mobile agents for real-world deployment.

Abstract

Vision-Language Models (VLMs) are increasingly deployed as autonomous agents to navigate mobile graphical user interfaces (GUIs). Operating in dynamic on-device ecosystems, which include notifications, pop-ups, and inter-app interactions, exposes them to a unique and underexplored threat vector: environmental injection. Unlike prompt-based attacks that manipulate textual instructions, environmental injection corrupts an agent's visual perception by inserting adversarial UI elements (for example, deceptive overlays or spoofed notifications) directly into the GUI. This bypasses textual safeguards and can derail execution, causing privacy leakage, financial loss, or irreversible device compromise. To systematically evaluate this threat, we introduce GhostEI-Bench, the first benchmark for assessing mobile agents under environmental injection attacks within dynamic, executable environments. Moving beyond static image-based assessments, GhostEI-Bench injects adversarial events into realistic application workflows inside fully operational Android emulators and evaluates performance across critical risk scenarios. We further propose a judge-LLM protocol that conducts fine-grained failure analysis by reviewing the agent's action trajectory alongside the corresponding screenshot sequence, pinpointing failure in perception, recognition, or reasoning. Comprehensive experiments on state-of-the-art agents reveal pronounced vulnerability to deceptive environmental cues: current models systematically fail to perceive and reason about manipulated UIs. GhostEI-Bench provides a framework for quantifying and mitigating this emerging threat, paving the way toward more robust and secure embodied agents.

Figures (4)

• Figure 1: GhostEI-Bench overview—environmental injection in dynamic on-device environments. GhostEI-Bench perturbs natural task flows through environmental injections (e.g., overlay attacks, popup SMS, implicit data leakage) to systematically evaluate agent safety and helpfulness across diverse application domains, risk categories, and multiple attack scenarios.
• Figure 2: Overview of GhostEI-Bench threat model and attack vectors (best viewed in color; zoom in for details). (a) Deceptive Instruction: probes foundational safety alignment by testing refusal to inherently harmful user prompts. (b) Static Environmental Injection: assesses contextual discretion when sensitive information is already present in the environment (e.g., passwords or financial details in notes/screenshots). (c) Dynamic Environmental Injection: evaluates robustness to real-time interruptions via system-style overlays or popup SMS that attempt to hijack actions.
• Figure 3: Distribution of 110 test cases across 7 application domains (left) and risk types (right). The left donut chart shows the frequency of each domain's involvement, with "Settings" (60) and "Productivity" (46) most common. The right bar chart details the risk distribution, highlighting "Privacy Leakage" (67) and "Fraud" (43) as the most common risks. Note that a single test case can cover multiple domains and risks, so the sum of counts in each chart is greater than 110.
• Figure 4: Attack success rate (%) of VLM agents, normalized within each vulnerability dimension. For each sub-plot, the y-axis indicates the proportion of tasks in that category (attack vector, risk type, or application domain) where attacks led to full or partial success, relative to the total number of tasks in that category. Results highlight three recurring failure modes: (i) dynamic environmental injection consistently achieves the highest success proportion across vectors, (ii) Fraud and Disinformation dominate among risk types, and (iii) Social Media and Life Services emerge as the most failure-prone domains. These findings reveal systematic patterns in how VLM agents are misled, providing a concrete basis for failure mode analysis.