Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agents

TL;DR

Mobile-Bench-v2 addresses key gaps in VLM-based mobile agent benchmarking by introducing slot-based instruction generation (GIAS), offline multi-path evaluation, and three evaluation splits (Common, Noisy, Ambiguous) to reflect real-world challenges like ads, pop-ups, and proactive information seeking. It combines the Mobile3M graph corpus with 12k generated instructions to enable stable reward signals and multi-path assessment, while also injecting realistic noise and enabling interactive Q&A. Empirical results show that different agent frameworks and large-language models have varying strengths across single- and multi-path settings, with multi-path evaluation offering a more faithful measure of robustness. The benchmark supports rigorous testing of noise-robustness and proactive interaction, and its open data and code foster broader development and evaluation of GUI agents in mobile contexts.

Abstract

VLM-based mobile agents are increasingly popular due to their capabilities to interact with smartphone GUIs and XML-structured texts and to complete daily tasks. However, existing online benchmarks struggle with obtaining stable reward signals due to dynamic environmental changes. Offline benchmarks evaluate the agents through single-path trajectories, which stands in contrast to the inherently multi-solution characteristics of GUI tasks. Additionally, both types of benchmarks fail to assess whether mobile agents can handle noise or engage in proactive interactions due to a lack of noisy apps or overly full instructions during the evaluation process. To address these limitations, we use a slot-based instruction generation method to construct a more realistic and comprehensive benchmark named Mobile-Bench-v2. Mobile-Bench-v2 includes a common task split, with offline multi-path evaluation to assess the agent's ability to obtain step rewards during task execution. It contains a noisy split based on pop-ups and ads apps, and a contaminated split named AITZ-Noise to formulate a real noisy environment. Furthermore, an ambiguous instruction split with preset Q\&A interactions is released to evaluate the agent's proactive interaction capabilities. We conduct evaluations on these splits using the single-agent framework AppAgent-v1, the multi-agent framework Mobile-Agent-v2, as well as other mobile agents such as UI-Tars and OS-Atlas. Code and data are available at https://huggingface.co/datasets/xwk123/MobileBench-v2.

Figures (13)

• Figure 1: The Mobile-Bench-v2 includes three types of tasks: Common-split, Noisy-split, and Ambiguous-split, and demonstrates the process of instruction generation and manual annotation for each task. In Noisy-split, the GUIs with red shading represent noise.
• Figure 2: Unlike Online Evaluation, offline multi-path evaluation checks both process and final GUI as reward signals.
• Figure 3: The task distribution chart is sorted by the number of simple tasks in descending order. The average steps for both simple and complex tasks in each app remain relatively balanced.
• Figure 4: Download volume distribution of music APPs in one month, on the left. The distribution of tasks and their respective step counts, in the middle. The full distribution of apps and their sampled distribution, on the right.
• Figure 5: The data distribution in Mobile3M.