Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AndroidGen: Building an Android Language Agent under Data Scarcity

Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shudan Zhang, Yuxiao Dong, Jie Tang

TL;DR

AndroidGen addresses data scarcity in mobile-agent learning by introducing a modular framework (ExpSearch, ReflectPlan, AutoCheck, StepCritic) that generates and evaluates trajectories on Android devices. It builds an open-source Android language agent by automatically collecting trajectories with GPT-4o, augmenting data via sub-goal trajectories, and fine-tuning GLM-4-9B and Llama-3-70B with LoRA. Extensive evaluation on AndroidWorld, Android in the Wild, and popular apps demonstrates improved reasoning, operation accuracy, and generalization, while offering a scalable, cost-efficient data-generation pipeline. The work provides practical tools and datasets to advance open-source, data-efficient Android agents and outlines avenues for further improvements and efficiency gains.

Abstract

Large language models have opened up a world of possibilities for various NLP tasks, sparking optimism for the future. Despite their potential, LLMs have yet to be widely used as agents on real mobile devices. The main challenge is the need for high-quality data sources. Time constraints and labor intensity often hinder human annotation. On the other hand, existing LLMs exhibit inadequate completion rates and need a robust data filtration strategy. Given these challenges, we develop a framework called AndroidGen to enhance the capabilities of LLM-based agents under data scarcity. In addition, we leverage AndroidGen to collect trajectories given human tasks and train open-source LLMs on these trajectories to develop an open-source mobile agent without manually labeled trajectories. We extensively evaluate AndroidGen with AndroidWorld, AitW, and various popular applications, demonstrating its improvements and revealing potential areas for future improvement. Code, model, and data are available at https://github.com/THUDM/AndroidGen.

AndroidGen: Building an Android Language Agent under Data Scarcity

TL;DR

AndroidGen addresses data scarcity in mobile-agent learning by introducing a modular framework (ExpSearch, ReflectPlan, AutoCheck, StepCritic) that generates and evaluates trajectories on Android devices. It builds an open-source Android language agent by automatically collecting trajectories with GPT-4o, augmenting data via sub-goal trajectories, and fine-tuning GLM-4-9B and Llama-3-70B with LoRA. Extensive evaluation on AndroidWorld, Android in the Wild, and popular apps demonstrates improved reasoning, operation accuracy, and generalization, while offering a scalable, cost-efficient data-generation pipeline. The work provides practical tools and datasets to advance open-source, data-efficient Android agents and outlines avenues for further improvements and efficiency gains.

Abstract

Large language models have opened up a world of possibilities for various NLP tasks, sparking optimism for the future. Despite their potential, LLMs have yet to be widely used as agents on real mobile devices. The main challenge is the need for high-quality data sources. Time constraints and labor intensity often hinder human annotation. On the other hand, existing LLMs exhibit inadequate completion rates and need a robust data filtration strategy. Given these challenges, we develop a framework called AndroidGen to enhance the capabilities of LLM-based agents under data scarcity. In addition, we leverage AndroidGen to collect trajectories given human tasks and train open-source LLMs on these trajectories to develop an open-source mobile agent without manually labeled trajectories. We extensively evaluate AndroidGen with AndroidWorld, AitW, and various popular applications, demonstrating its improvements and revealing potential areas for future improvement. Code, model, and data are available at https://github.com/THUDM/AndroidGen.
Paper Structure (42 sections, 15 figures, 10 tables, 1 algorithm)

This paper contains 42 sections, 15 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. AndroidGen Framework
  4. Environment
  5. Observation Space
  6. Action Space
  7. Agent Architecture
  8. ExpSearch
  9. ReflectPlan
  10. AutoCheck
  11. StepCritic
  12. Building An Open-Source Android Language Agent
  13. Data Collection
  14. Training
  15. Experiments
  16. ...and 27 more sections

Figures (15)

  • Figure 1: The success rates of popular mobile agents and humans on AndroidWorld.
  • Figure 2: Examples of AndroidGen's execution on four user tasks.
  • Figure 3: Overview of AndroidGen framework designed to complete tasks in Android. Our process comprises three stages: preliminary, task execution, and update. Preliminary (a): ExpSearch retrieve the top-1 similar tasks and trajectories from the database and feed them into the agent. Task Execution (b): ReflectPlan assesses the progress and updates the plan. Then, the agent generates operations based on the environment, plan, and retrieval example. AutoCheck verifies these operations, executing them if successful or regenerating them if not. Update (c): StepCritic evaluates the trajectories in fine-grand and updates the database accordingly.
  • Figure 4: AndroidGen data construction workflow.
  • Figure 5: Audio Recorder
  • ...and 10 more figures