Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents

Yue Cao, Yingyao Wang, Pi Bu, Jingxuan Xing, Wei Jiang, Zekun Zhu, Junpeng Ma, Sashuai Zhou, Tong Lu, Jun Song, Yu Cheng, Yuning Jiang, Bo Zheng

TL;DR

AndroidLens presents a large, bilingual benchmark (571 tasks across 38 domains) for long-latency Android GUI agents, emphasizing realistic, multi-step interactions and cross-app behaviors. It introduces static and dynamic milestone-based evaluation, including the Average Task Progress (ATP) metric, to provide fine-grained, bias-reduced assessments beyond final outcomes. Across 15 evaluated agents, performance remains limited (best SR ~12.7% dynamic; ATP ~50%), underscoring challenges in widget-level perception, memory across app switches, and robust error recovery. The work also details a rigorous data-curation pipeline and a decoupled evaluation framework to enable fair, scalable benchmarking and future improvements in mobile GUI intelligence.

Abstract

Graphical user interface (GUI) agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term memory retention.

AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents

TL;DR

AndroidLens presents a large, bilingual benchmark (571 tasks across 38 domains) for long-latency Android GUI agents, emphasizing realistic, multi-step interactions and cross-app behaviors. It introduces static and dynamic milestone-based evaluation, including the Average Task Progress (ATP) metric, to provide fine-grained, bias-reduced assessments beyond final outcomes. Across 15 evaluated agents, performance remains limited (best SR ~12.7% dynamic; ATP ~50%), underscoring challenges in widget-level perception, memory across app switches, and robust error recovery. The work also details a rigorous data-curation pipeline and a decoupled evaluation framework to enable fair, scalable benchmarking and future improvements in mobile GUI intelligence.

Abstract

Graphical user interface (GUI) agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term memory retention.
Paper Structure (44 sections, 13 figures, 8 tables)

This paper contains 44 sections, 13 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mobile GUI Agents
  4. Mobile GUI Benchmarks
  5. AndroidLens Task
  6. Task Definition
  7. Data Curation Pipeline
  8. Task Construction.
  9. Trajectory Annotation.
  10. Quality Control.
  11. Dataset Statistics
  12. Milestone-Based Task Evaluation
  13. Static Evaluation
  14. Evaluation Metrics.
  15. Multi-trajectory Evaluation.
  16. ...and 29 more sections

Figures (13)

  • Figure 1: An overview of the AndroidLens benchmark. Its main characteristics are: (1) It covers long-latency tasks with multiple constraints, multiple objectives, and domain-specific requirements, involving increasingly complex widget operations, cross-app interactions, batch operations, and more. (2) It adopts a milestone-based intermediate-goal evaluation scheme. By using stable and verifiable milestones to measure average task progress, the benchmark provides fine-grained result analysis for long-latency tasks.
  • Figure 2: Data curation pipeline of AndroidLens. The pipeline includes Task Construction, Trajectory Annotation and Quality Control.
  • Figure 3: Comparison of trajectory lengths between AndroidLens and existing benchmarks.
  • Figure 4: Action matching scores of high-level instructions across different task categories. The range of all axes is normalized to 0-30.
  • Figure 5: Average task progress curves of agents under different step lengths.
  • ...and 8 more figures