AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation

TL;DR

AutoDroid-V2 reframes mobile GUI task automation as on-device code generation guided by a fine-grained app document, enabling script-based execution with strong privacy and efficiency benefits. It introduces offline app document generation (state grouping, abstract elements, and ETG dependencies) and online data synthesis plus a runtime interpreter with dependency-aware execution and prompt caching. Through extensive experiments on DroidTask and AitW-subset, it demonstrates higher task success rates, substantially lower latency, and reduced token costs compared with prior step-wise and on-device baselines, all while maintaining robust performance across model sizes. The work highlights the potential of document-guided scripting for scalable, on-device GUI agents with practical privacy and cost advantages, and discusses limitations and avenues for future integration with step-wise reasoning and vision-based grounding.

Abstract

Large language models (LLMs) have brought exciting new advances to mobile UI agents, a long-standing research field that aims to complete arbitrary natural language tasks through mobile UI interactions. However, existing UI agents usually demand powerful large language models that are difficult to be deployed locally on end-users' devices, raising huge concerns about user privacy and centralized serving cost. Inspired by the remarkable coding abilities of recent small language models (SLMs), we propose to convert the UI task automation problem to a code generation problem, which can be effectively solved by an on-device SLM and efficiently executed with an on-device code interpreter. Unlike normal coding tasks that can be extensively pre-trained with public datasets, generating UI automation code is challenging due to the diversity, complexity, and variability of target apps. Therefore, we adopt a document-centered approach that automatically builds fine-grained API documentation for each app and generates diverse task samples based on this documentation. By guiding the agent with the synthetic documents and task samples, it learns to generate precise and efficient scripts to complete unseen tasks. Based on detailed comparisons with state-of-the-art mobile UI agents, our approach effectively improves the mobile task automation with significantly higher success rates and lower latency/token consumption. Code is open-sourced at https://github.com/MobileLLM/AutoDroid-V2.

Figures (5)

• Figure 1: Comparison of conventional step-wise GUI agent and our script-based GUI agent. The step-wise GUI agent includes the user’s task and the current GUI state in the LLM prompt ($Q1, \dots Qn$), asking the model to generate a GUI action for each step ($A1, \dots An$). The script-based agent inputs the task and app document ($Q$), prompting the LLM to generate a multi-step script for execution.
• Figure 2: The architecture of AutoDroid-V2.
• Figure 3: The document of an app containing essential abstract elements in one GUI.
• Figure 4: Average LLM inference latency of AutoDroid-V2 and baseline (AutoDroid) on DroidTask.
• Figure 5: Success rate and reversed redundancy ratio of AutoDroid-V2 and baseline based on different LLMs. llama3.2-3b, qwen2.5-7b, llama3.1-8b