LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects

TL;DR

This survey analyzes the emergence of LLM-powered phone GUI agents that bridge natural language understanding with GUI perception and on-device action. It introduces a mobile-centric taxonomy of frameworks (single, multi-agent, plan-then-act), modeling approaches (prompt engineering and training-based methods), and resources (datasets/benchmarks), while highlighting on-device deployment and security considerations. The authors identify core challenges—generalization gaps, maintenance costs, intent understanding, and GUI perception—and explain how LLMs, multimodal grounding, planning, and reflection address them. They provide a critical appraisal of datasets and benchmarks, discuss performance trends (notably RL gains offline versus online deployment), and offer a forward-looking roadmap emphasizing hybrid training pipelines, richer evaluation, and robust safety and privacy mechanisms. Collectively, the paper guides researchers and practitioners toward scalable, user-friendly, and secure mobile GUI agents through standardized frameworks, curated datasets, and rigorous evaluation protocols.

Abstract

With the rapid rise of large language models (LLMs), phone automation has undergone transformative changes. This paper systematically reviews LLM-driven phone GUI agents, highlighting their evolution from script-based automation to intelligent, adaptive systems. We first contextualize key challenges, (i) limited generality, (ii) high maintenance overhead, and (iii) weak intent comprehension, and show how LLMs address these issues through advanced language understanding, multimodal perception, and robust decision-making. We then propose a taxonomy covering fundamental agent frameworks (single-agent, multi-agent, plan-then-act), modeling approaches (prompt engineering, training-based), and essential datasets and benchmarks. Furthermore, we detail task-specific architectures, supervised fine-tuning, and reinforcement learning strategies that bridge user intent and GUI operations. Finally, we discuss open challenges such as dataset diversity, on-device deployment efficiency, user-centric adaptation, and security concerns, offering forward-looking insights into this rapidly evolving field. By providing a structured overview and identifying pressing research gaps, this paper serves as a definitive reference for researchers and practitioners seeking to harness LLMs in designing scalable, user-friendly phone GUI agents. The collection of papers reviewed in this survey will be hosted and regularly updated on the GitHub repository: https://github.com/PhoneLLM/Awesome-LLM-Powered-Phone-GUI-Agents

Figures (11)

• Figure 1: Comparison between conversational LLMs and phone GUI agents. While a conversational LLM can understand queries and provide informative responses (e.g., recommending coffee beans), a Phone GUI agent can go beyond text generation to perceive the device's interface, decide on an appropriate action (like tapping an app icon), and execute it in the real environment, thus enabling tasks like ordering a latte directly on the user's phone.
• Figure 2: A comprehensive taxonomy of LLM-powered phone GUI agents in phone automation. Note that only a selection of representative works is included in this categorization.
• Figure 3: Milestones in the development of LLM-powered phone GUI agents. This figure divides advancements into four primary parts: Prompt Engineering, Training-Based Methods, Datasets and Benchmarks. Prompt Engineering leverages pre-trained LLMs by strategically crafting input prompts, as detailed in §\ref{['subsec:prompt_engineering']}, to perform specific tasks without modifying model parameters. In contrast, Training-Based Methods, discussed in §\ref{['subsec:training_based']}, involve adapting LLMs via supervised fine-tuning or reinforcement learning on GUI-specific data, thereby enhancing their ability to understand and interact with mobile UIs.
• Figure 4: POMDP model for ordering a latte. Each circle represents a state (e.g., Home Screen, App Homepage, Latte Details Page, Customize Order, Order Confirmation, Order Complete). The agent starts at the initial state $S_0$ (Home Screen) and makes decisions at each step (e.g., tapping the Starbucks app icon, selecting the "Latte" button, viewing latte details). Due to partial observability, the agent receives limited information at each decision point (e.g., $O_0$: Starbucks app icon visible, $O_1$: "Latte" button visible, $O_2$: Latte product details visible). Some actions correctly advance towards the goal, while others may cause errors requiring corrections. The final goal is to confirm the order.
• Figure 5: Overview of MLLM-powered phone GUI agent framework. The user's intent, expressed through natural language, is mapped to UI operations. By perceiving UI information and phone state(§\ref{['subsec:perception']}) , the agent leverages stored knowledge and memory to plan, reason, and reflect (§\ref{['subsec:brain']}) . Finally, it executes actions to fulfill the user's goals(§\ref{['subsec:action']}).