MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment

TL;DR

MobileBench-OL introduces a comprehensive online benchmark for evaluating mobile GUI agents in real-world environments, combining 1080 tasks across 80 Chinese apps and five subsets that assess basic capabilities, long-horizon reasoning, GUI exploration, and noise robustness. It pairs a data-construction pipeline with an Auto-Eval framework and a Reset Mechanism to enable stable, repeatable evaluations on real devices. Evaluations of 12 GUI agents show meaningful gaps to real-world requirements, with clear insights into where reasoning, exploration, and robustness can improve. The benchmark offers a scalable, fair platform that can drive progress in real-world GUI agent capabilities and evaluation methodology.

Abstract

Recent advances in mobile Graphical User Interface (GUI) agents highlight the growing need for comprehensive evaluation benchmarks. While new online benchmarks offer more realistic testing than offline ones, they tend to focus on the agents' task instruction-following ability while neglecting their reasoning and exploration ability. Moreover, these benchmarks do not consider the random noise in real-world mobile environments. This leads to a gap between benchmarks and real-world environments. To addressing these limitations, we propose MobileBench-OL, an online benchmark with 1080 tasks from 80 Chinese apps. It measures task execution, complex reasoning, and noise robustness of agents by including 5 subsets, which set multiple evaluation dimensions. We also provide an auto-eval framework with a reset mechanism, enabling stable and repeatable real-world benchmarking. Evaluating 12 leading GUI agents on MobileBench-OL shows significant room for improvement to meet real-world requirements. Human evaluation further confirms that MobileBench-OL can reliably measure the performance of leading GUI agents in real environments. Our data and code will be released upon acceptance.

Figures (21)

• Figure 1: In addition to instruction-following ability, MobileBench-OL also measures long-horizon reasoning, exploration, and real-world noise handling.
• Figure 2: Examples of three exploration abilities (above) and four noise types (bottom).
• Figure 3: Data construction pipeline.
• Figure 4: Pipeline of the Auto-Eval framework.
• Figure 5: Online platform interface.