Practical and Stealthy Touch-Guided Jailbreak Attacks on Deployed Mobile Vision-Language Agents

TL;DR

This work investigates practical, stealthy jailbreaks for LVLM-driven mobile agents by embedding a compact in-app prompt that is selectively revealed during agent perception. It introduces HG-IDA*, a two-stage detoxification-based one-shot jailbreak comprising template design and keyword-level perturbations to bypass safety filters while preserving intent. The framework relies on non-privileged in-app embedding and agent-attributable activation, evaluated across three Android apps and multiple LVLM backends, revealing high planning hijack ($T_{asr}$) and execution hijack ($R_{asr}$) rates on several backends (e.g., GPT-4o $T_{asr}=75.0\%$, $R_{asr}=66.7\%$; Gemini-2.0-pro $T_{asr}=95.0\%$, $R_{asr}=82.5\%$). A defense based on provenance-aware prompting shows substantial mitigation, reducing attack success to single-digit percentages, underscoring the need for robust, input-origin attribution in mobile agents. Overall, the paper exposes a practical security vulnerability in deployed mobile vision-language agents and offers concrete methods for both attack and defense in real-world settings.

Abstract

Large vision-language models (LVLMs) enable autonomous mobile agents to operate smartphone user interfaces, yet vulnerabilities in their perception and interaction remain critically understudied. Existing research often relies on conspicuous overlays, elevated permissions, or unrealistic threat assumptions, limiting stealth and real-world feasibility. In this paper, we introduce a practical and stealthy jailbreak attack framework, which comprises three key components: (i) non-privileged perception compromise, which injects visual payloads into the application interface without requiring elevated system permissions; (ii) agent-attributable activation, which leverages input attribution signals to distinguish agent from human interactions and limits prompt exposure to transient intervals to preserve stealth from end users; and (iii) efficient one-shot jailbreak, a heuristic iterative deepening search algorithm (HG-IDA*) that performs keyword-level detoxification to bypass built-in safety alignment of LVLMs. Moreover, we developed three representative Android applications and curated a prompt-injection dataset for mobile agents. We evaluated our attack across multiple LVLM backends, including closed-source services and representative open-source models, and observed high planning and execution hijack rates (e.g., GPT-4o: 82.5% planning / 75.0% execution), exposing a fundamental security vulnerability in current mobile agents and underscoring critical implications for autonomous smartphone operation.

Figures (4)

• Figure 1: A real-world example of our privacy-leakage attack on mobile agents using GPT-4o. A malicious prompt is pre-embedded in the app and briefly revealed for 30 seconds when the agent interacts with the interface, corrupting the agent’s perception and causing it to exfiltrate private user data. The attacker then receives an email from the agent containing the user’s private information, posing a severe security threat.
• Figure 2: Comparison of Thought ASR ($T_{asr}$,%) and Result ASR ($R_{asr}$,%) across evaluated multimodal backends. Each pair of bars shows the percentage of successful planning hijacks ($T_{asr}$, left) and end-to-end execution ($R_{asr}$, right); annotated values highlight models with large $T_{asr}$–$R_{asr}$ gaps versus those vulnerable at both stages.
• Figure 3: Example workflow of a stealthy in-app prompt injection that compromises a mobile agent. An attacker pre-embeds a short malicious prompt inside the app UI which remains hidden during normal use and is selectively revealed only under automated (ADB-driven) interaction; the disclosure follows a three-step trigger sequence: (1) trigger the previous page, (2) trigger the page, (3) agent execution, causing the agent to incorporate the injected instruction into its plan and perform the attacker’s action. Note: All example data shown in this figure are synthetic and redacted.
• Figure 4: Examples of prompt-injection attacks across our malicious apps. Subfigure (a) shows an injection in the memo app; (b) shows an injection in the smart-home app; (c) shows a social app injection whose command appears innocuous (benign-looking) but nevertheless induces harmful agent behavior; and (d) shows a social app injection with an explicitly harmful command. Note: All example data shown in this figure are synthetic and redacted.