Zero-Permission Manipulation: Can We Trust Large Multimodal Model Powered GUI Agents?

TL;DR

This work exposes Action Rebinding, a temporal exploit that hijacks LMM-powered GUI agents on Android by exploiting the observation-to-action gap. It formalizes the attack through three primitives—Atomic Action Rebinding, Multi-step Orchestration, and Intent Alignment Strategy—and demonstrates near-ubiquitous success across six open-source agents on 15 tasks, including zero-permission, stealthy execution. The study reveals that Android's UI state preservation and foreground-transition mechanisms enable programmable attack chains without privileged permissions, challenging the efficacy of traditional defenses. The findings underscore a fundamental architectural vulnerability in agent-OS integration and call for new defenses ensuring context integrity across perception and execution for autonomous GUI agents.

Abstract

Large multimodal model powered GUI agents are emerging as high-privilege operators on mobile platforms, entrusted with perceiving screen content and injecting inputs. However, their design operates under the implicit assumption of Visual Atomicity: that the UI state remains invariant between observation and action. We demonstrate that this assumption is fundamentally invalid in Android, creating a critical attack surface. We present Action Rebinding, a novel attack that allows a seemingly-benign app with zero dangerous permissions to rebind an agent's execution. By exploiting the inevitable observation-to-action gap inherent in the agent's reasoning pipeline, the attacker triggers foreground transitions to rebind the agent's planned action toward the target app. We weaponize the agent's task-recovery logic and Android's UI state preservation to orchestrate programmable, multi-step attack chains. Furthermore, we introduce an Intent Alignment Strategy (IAS) that manipulates the agent's reasoning process to rationalize UI states, enabling it to bypass verification gates (e.g., confirmation dialogs) that would otherwise be rejected. We evaluate Action Rebinding Attacks on six widely-used Android GUI agents across 15 tasks. Our results demonstrate a 100% success rate for atomic action rebinding and the ability to reliably orchestrate multi-step attack chains. With IAS, the success rate in bypassing verification gates increases (from 0% to up to 100%). Notably, the attacker application requires no sensitive permissions and contains no privileged API calls, achieving a 0% detection rate across malware scanners (e.g., VirusTotal). Our findings reveal a fundamental architectural flaw in current agent-OS integration and provide critical insights for the secure design of future agent systems. To access experimental logs and demonstration videos, please contact yi_qian@smail.nju.edu.cn.

Figures (9)

• Figure 1: The agent's action to send a message is misbound to answering the phone due to an unexpected phone call.
• Figure 2: An example of rebinding a tap to purchase. The agent plans to tap the new notes button in the NOTES App based on its observation. During the time window between the agent's thinking and execution, the NOTES App wakes up a Shopping App. Unaware that the app has changed, the agent executes the tap, causing the action to be misbound to an unauthorized purchase.
• Figure 3: Workflow of action rebinding attack.
• Figure 4: The action planned for clicking the back button is rebound to click the notification.
• Figure 5: Comparison of carrier with (right) and without (left) IAS. Without IAS, the agent detects a context mismatch and rejects the verification. With IAS, the agent rationalizes and accepts the verification.