ProBench: Benchmarking GUI Agents with Accurate Process Information

TL;DR

ProBench addresses a critical gap in GUI agent evaluation by introducing a Process Provider that delivers accurate process information for Process-related tasks, enabling automatic, process-aware benchmarking across 200+ tasks in 34 bilingual apps. The benchmark combines Task Curation, a Dynamic Environment, and an Evaluation Pipeline to measure both final-state accuracy and intermediate operation quality, revealing that even strong models struggle in real-world GUI scenarios, especially for social/lifestyle apps. A clear scaling effect is observed across model families, yet universal deficiencies in grounding, historical operation attention, and multi-step task planning persist. By establishing a rigorous, scalable standard for process-aware GUI evaluation, ProBench offers a practical path toward more capable and reliable GUI agents.

Abstract

With the deep integration of artificial intelligence and interactive technology, Graphical User Interface (GUI) Agent, as the carrier connecting goal-oriented natural language and real-world devices, has received widespread attention from the community. Contemporary benchmarks aim to evaluate the comprehensive capabilities of GUI agents in GUI operation tasks, generally determining task completion solely by inspecting the final screen state. However, GUI operation tasks consist of multiple chained steps while not all critical information is presented in the final few pages. Although a few research has begun to incorporate intermediate steps into evaluation, accurately and automatically capturing this process information still remains an open challenge. To address this weakness, we introduce ProBench, a comprehensive mobile benchmark with over 200 challenging GUI tasks covering widely-used scenarios. Remaining the traditional State-related Task evaluation, we extend our dataset to include Process-related Task and design a specialized evaluation method. A newly introduced Process Provider automatically supplies accurate process information, enabling presice assessment of agent's performance. Our evaluation of advanced GUI agents reveals significant limitations for real-world GUI scenarios. These shortcomings are prevalent across diverse models, including both large-scale generalist models and smaller, GUI-specific models. A detailed error analysis further exposes several universal problems, outlining concrete directions for future improvements.

Figures (11)

• Figure 1: Illustration of a false outcome under existing evaluation. Red border indicates the ignored critical action.
• Figure 2: Overview of our ProBench benchmark. ProBench is a comprehensive mobile benchmark, comprising three key modules: (i) Task Curation: We select 34 mainstream bilingual applications, generate candidate tasks using LLMs, and refine them through manual review. (ii) Dynamic Environment: Agents complete the specified tasks by controlling the device. (iii) Evaluation Pipeline: For Process-related Task, we optionally choose either the Structure Description Converter or the MLLM-based Summarizer of Process Provider to supply process information. The final evaluation is performed by the judger selected from the judger group.
• Figure 3: Statistics of ProBench. The left illustrates the diverse application categories included. The right displays the amount of each type tasks within different languages.
• Figure 5: Accuracies of GUI agents on ProBench among different application categories. We demonstrate the best proprietary, general open-source and GUI-specific models.
• Figure 6: The lack of grounding capability in GUI agents. We show the accurate screen location where the agent actually clicked after coordinate conversion.