Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A3: Android Agent Arena for Mobile GUI Agents

Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guangyi Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, Hongsheng Li

TL;DR

<3-5 sentence high-level summary> Android Agent Arena (A3) introduces a dynamic, autonomous evaluation platform for mobile GUI agents. It combines a real-time Appium-based pipeline with an expanded action space and 201 tasks across 20 mainstream apps, enabling both function-based and LLM-based automated evaluation. The authors show that GUI-task-tuned agents outperform generalist LLMs, and that essential-state based ESAR metrics provide finer-grained insight, with LLM evaluators aligning closely to human judgments. The work addresses limitations of static-frame datasets and simplistic dynamic benchmarks, offering a scalable, real-world benchmarking framework for mobile GUI agents and outlining practical considerations for deployment and future improvements.

Abstract

AI agents have become increasingly prevalent in recent years, driven by significant advancements in the field of large language models (LLMs). Mobile GUI agents, a subset of AI agents, are designed to autonomously perform tasks on mobile devices. While numerous studies have introduced agents, datasets, and benchmarks to advance mobile GUI agent research, many existing datasets focus on static frame evaluations and fail to provide a comprehensive platform for assessing performance on real-world, in-the-wild tasks. To address this gap, we present Android Agent Arena (A3), a novel evaluation platform. Unlike existing in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as real-time online information retrieval and operational instructions; (2) a larger, more flexible action space, enabling compatibility with agents trained on any dataset; and (3) automated business-level LLM-based evaluation process. A3 includes 21 widely used general third-party apps and 201 tasks representative of common user scenarios, providing a robust foundation for evaluating mobile GUI agents in real-world situations and a new autonomous evaluation process for less human labor and coding expertise. The project is available at https://yuxiangchai.github.io/Android-Agent-Arena/.

A3: Android Agent Arena for Mobile GUI Agents

TL;DR

<3-5 sentence high-level summary> Android Agent Arena (A3) introduces a dynamic, autonomous evaluation platform for mobile GUI agents. It combines a real-time Appium-based pipeline with an expanded action space and 201 tasks across 20 mainstream apps, enabling both function-based and LLM-based automated evaluation. The authors show that GUI-task-tuned agents outperform generalist LLMs, and that essential-state based ESAR metrics provide finer-grained insight, with LLM evaluators aligning closely to human judgments. The work addresses limitations of static-frame datasets and simplistic dynamic benchmarks, offering a scalable, real-world benchmarking framework for mobile GUI agents and outlining practical considerations for deployment and future improvements.

Abstract

AI agents have become increasingly prevalent in recent years, driven by significant advancements in the field of large language models (LLMs). Mobile GUI agents, a subset of AI agents, are designed to autonomously perform tasks on mobile devices. While numerous studies have introduced agents, datasets, and benchmarks to advance mobile GUI agent research, many existing datasets focus on static frame evaluations and fail to provide a comprehensive platform for assessing performance on real-world, in-the-wild tasks. To address this gap, we present Android Agent Arena (A3), a novel evaluation platform. Unlike existing in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as real-time online information retrieval and operational instructions; (2) a larger, more flexible action space, enabling compatibility with agents trained on any dataset; and (3) automated business-level LLM-based evaluation process. A3 includes 21 widely used general third-party apps and 201 tasks representative of common user scenarios, providing a robust foundation for evaluating mobile GUI agents in real-world situations and a new autonomous evaluation process for less human labor and coding expertise. The project is available at https://yuxiangchai.github.io/Android-Agent-Arena/.
Paper Structure (27 sections, 4 figures, 4 tables)

This paper contains 27 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agent
  4. GUI-related Dataset
  5. Dynamic Evaluation Benchmark
  6. Android Agent Arena (A3)
  7. Overview
  8. Action Space
  9. Task
  10. Evaluation
  11. Evaluation Function
  12. LLM Evaluation System
  13. Experiments
  14. Evaluation
  15. Error Case Analysis
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Overview of Android Agent Arena. A3 contains 201 tasks from 20 widely used apps. Tasks are categorized into operation, single-frame query and multi-frame query based on the task goal. Tasks are also split into three difficulty levels based on the human operation steps. A3 also integrates two evaluation methods for different use cases.
  • Figure 2: Overview of Android Agent Arena. A3 contains controller, evaluator, and translator. The controller is responsible for controlling and getting the state of the device. The translator is responsible for translating the device function and the agent messages. The evaluator is responsible for the final evaluation.
  • Figure 3: Step 1 and Step 2 are correct, however, the agent starts typing before the search bar is clicked or selected, so the process sticks at this situation and the agent keeps typing and waiting.
  • Figure 4: Step 1 and Step 2 are correct, however, the agent predicts a wrong click coordinate and accidentally go to the shopping cart. It should go back but seems it lacks the capability to do that and gets stuck in the shopping cart.