AppAgentX: Evolving GUI Agents as Proficient Smartphone Users

TL;DR

The paper tackles inefficiencies in LLM-based GUI agents by introducing AppAgentX, a memory-augmented framework that evolves high-level actions from task history. It employs a chain-based memory to capture page/element interactions and learns shortcut nodes that compress repetitive low-level sequences, enabling faster and more accurate task execution on visual interfaces without backend APIs. Across multiple benchmarks, AppAgentX outperforms baselines in both efficiency and accuracy, with ablations confirming the value of memory and the expanded action space. The approach promises practical impact for on-device GUI automation and sets the stage for open-source development.

Abstract

Recent advancements in Large Language Models (LLMs) have led to the development of intelligent LLM-based agents capable of interacting with graphical user interfaces (GUIs). These agents demonstrate strong reasoning and adaptability, enabling them to perform complex tasks that traditionally required predefined rules. However, the reliance on step-by-step reasoning in LLM-based agents often results in inefficiencies, particularly for routine tasks. In contrast, traditional rule-based systems excel in efficiency but lack the intelligence and flexibility to adapt to novel scenarios. To address this challenge, we propose a novel evolutionary framework for GUI agents that enhances operational efficiency while retaining intelligence and flexibility. Our approach incorporates a memory mechanism that records the agent's task execution history. By analyzing this history, the agent identifies repetitive action sequences and evolves high-level actions that act as shortcuts, replacing these low-level operations and improving efficiency. This allows the agent to focus on tasks requiring more complex reasoning, while simplifying routine actions. Experimental results on multiple benchmark tasks demonstrate that our approach significantly outperforms existing methods in both efficiency and accuracy. The code will be open-sourced to support further research.

Figures (7)

• Figure 1: Overview of the LLM-based GUI agent baseline. At each step, the agent captures the current screen of the device and analyzes the interface to select an appropriate action from the predefined action space. The chosen action is then executed to interact with the GUI.
• Figure 2: Overview of the proposed framework. (a) The trajectory of a task execution is decomposed into multiple overlapping triples. Based on these triples, the LLM generates functional descriptions of both pages and UI elements. Descriptions of pages that are repeatedly generated are then merged. The entire interaction history is recorded using a chain of nodes. (b) The proposed evolutionary mechanism and execution process. The evolutionary mechanism generates shortcut nodes, which allow the agent to execute a series of actions efficiently without reasoning step by step. (c) An example of the content of nodes within the chain. Each node records essential information, including descriptions of pages, UI elements, and high-level actions, to facilitate understanding of the agent’s interactions.
• Figure 3: Comparison of Average Execution Time per Step. This figure presents the average execution time per steps across different LLMs and frameworks.
• Figure 4: Task Completion Times Across Different Task Lengths. This figure shows the distribution of task completion times for short, medium, and long tasks. Each violin plot represents the density of completion times, with wider sections indicating higher data concentration. AppAgentX consistently outperforms the baseline, particularly for longer tasks.
• Figure 5: Qualitative Results. This figure presents the path execution utilizing shortcuts on Gmail. It demonstrates the efficiency of AppAgentX in decreasing the utilization of LLM for per-step reasoning.