GraphPilot: GUI Task Automation with One-Step LLM Reasoning Powered by Knowledge Graph

TL;DR

GraphPilot addresses the latency and reliability issues of LLM-powered mobile GUI automation by constructing per-app knowledge graphs that encode page and element functions plus explicit transition rules. In the online phase, the knowledge graph steers LLM reasoning to generate a complete action sequence in almost one query, with a dynamic HTML fallback and a validator to enforce correctness. Tested on the DroidTask benchmark, GraphPilot delivers a higher task completion rate (74.1%) and substantial latency reductions relative to Mind2Web and AutoDroid, demonstrating that embedding structured domain knowledge in prompts can enable near one-shot planning for complex GUI tasks. This approach suggests a practical path toward faster, more reliable AI-powered personal assistants for mobile devices.

Abstract

Mobile graphical user interface (GUI) agents are designed to automate everyday tasks on smartphones. Recent advances in large language models (LLMs) have significantly enhanced the capabilities of mobile GUI agents. However, most LLM-powered mobile GUI agents operate in stepwise query-act loops, which incur high latency due to repeated LLM queries. We present GraphPilot, a mobile GUI agent that leverages knowledge graphs of the target apps to complete user tasks in almost one LLM query. GraphPilot operates in two complementary phases to enable efficient and reliable LLM-powered GUI task automation. In the offline phase, it explores target apps, records and analyzes interaction history, and constructs an app-specific knowledge graph that encodes functions of pages and elements as well as transition rules for each app. In the online phase, given an app and a user task, it leverages the knowledge graph of the given app to guide the reasoning process of LLM. When the reasoning process encounters uncertainty, GraphPilot dynamically requests the HTML representation of the current interface to refine subsequent reasoning. Finally, a validator checks the generated sequence of actions against the transition rules in the knowledge graph, performing iterative corrections to ensure it is valid. The structured, informative information in the knowledge graph allows the LLM to plan the complete sequence of actions required to complete the user task. On the DroidTask benchmark, GraphPilot improves task completion rate over Mind2Web and AutoDroid, while substantially reducing latency and the number of LLM queries.

Figures (7)

• Figure 1: A simple example of the process of the stepwise GUI agents handling the task "start the stopwatch" in the Clock app.
• Figure 2: Overview of GraphPilot.
• Figure 3: An example of the knowledge graph for an app.
• Figure 4: A simple example of the prompt generated by the prompt generator. The gray part represents a fixed background description. The red part represents dynamic content. The green part shows an example output generated by the LLM. (X, A) in the knowledge graph represents that interacting with element X will jump to page A.
• Figure 5: TCR in different numbers of actions required to complete the task across different GUI agents.