LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark

TL;DR

LearnAct introduces a demonstration-based paradigm for mobile GUI agents to address generalization in unseen apps. It pairs LearnGUI, a benchmark with offline and online tasks, with a three-agent framework (DemoParser, KnowSeeker, ActExecutor) to extract, retrieve, and apply demonstration knowledge during task execution. Experiments show substantial offline gains (e.g., Gemini-1.5-Pro increasing from 19.3% to 51.7% with a single demonstration) and online gains (UI-TARS-7B-SFT rising from 18.1% to 32.8%), demonstrating the practical viability of demonstrations for personalization and deployability. The work provides a systematic dataset and framework to study few-shot learning in dynamic mobile GUI environments, with implications for building adaptable, user-specific automation solutions.

Abstract

Mobile GUI agents show promise in automating tasks but face generalization challenges in diverse real-world scenarios. Traditional approaches using pre-training or fine-tuning with massive datasets struggle with the diversity of mobile applications and user-specific tasks. We propose enhancing mobile GUI agent capabilities through human demonstrations, focusing on improving performance in unseen scenarios rather than pursuing universal generalization through larger datasets. To realize this paradigm, we introduce LearnGUI, the first comprehensive dataset specifically designed for studying demonstration-based learning in mobile GUI agents, comprising 2,252 offline tasks and 101 online tasks with high-quality human demonstrations. We further develop LearnAct, a sophisticated multi-agent framework that automatically extracts knowledge from demonstrations to enhance task completion. This framework integrates three specialized agents: DemoParser for knowledge extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for demonstration-enhanced task execution. Our experimental results show significant performance gains in both offline and online evaluations. In offline assessments, a single demonstration improves model performance, increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from 18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish demonstration-based learning as a promising direction for more adaptable, personalized, and deployable mobile GUI agents.

Figures (15)

• Figure 1: A toy example for demonstration learning on mobile GUI Agent. We build a benchmark named LearnGUI for demonstration learning on Mobile GUI Agent, which provides different few-shot task combinations and offers multi-dimensional metrics including task similarity, UI similarity, and action similarity between support tasks and query tasks.
• Figure 2: Joint distribution of UI similarity and action similarity in LearnGUI-Offline. The scatter plot shows the relationship between UI and action similarity measures across task pairs. The quadrant divisions represent our categorization of tasks into four profiles: UISHActSH, UISHActSL, UISLActSH, and UISLActSL, enabling analysis of how different similarity combinations affect learning transfer.
• Figure 3: Illustration of the overall framework of LearnAct. Architecture diagram showing the three main components (DemoParser, KnowSeeker, ActExecutor) and their interconnections within the LearnAct system, including data flow from human demonstrations to execution.
• Figure 4: Pipeline of DemoParser Agent. Input instructions and corresponding actions and screenshots; output low-level action descriptions and create knowledge database. This process transforms high-level user instructions into precise operation sequences while building a reusable domain knowledge base to improve mobile interface interaction automation efficiency.
• Figure 5: Pipeline of KnowSeeker Agent. The KnowSeeker Agent converts demo trajectories from the knowledge base into a vector database. When executing user tasks, KnowSeeker retrieves the top-k relevant demos from the vector database for subsequent use. This approach enables efficient retrieval of similar demonstrations to assist with new task execution.