Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

Yang Li, Yuchen Liu, Haoyu Lu, Zhiqiang Xia, Hongzhen Wang, Kaiyang Han, Changpeng Yang, Jinyang Wu, Jiaming Xu, Runyu Shi, Ying Huang

Abstract

Recent progress in Multimodal Large Language Models (MLLMs) has enabled mobile GUI agents capable of visual perception, cross-modal reasoning, and interactive control. However, existing benchmarks are largely English-centric and fail to capture the linguistic and interaction characteristics of the Chinese mobile ecosystem. They also focus on isolated skills such as GUI grounding or offline agent, lacking a unified and fine-grained framework to assess the full capability chain from perception to execution. To address this gap, we introduce GUI-CEval, the first comprehensive benchmark for Chinese mobile GUI agents, built entirely on physical device environments. GUI-CEval spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation. All data are collected and verified through multi-stage manual processes to ensure authenticity and reproducibility. Extensive experiments on 20 representative MLLMs and multi-agent systems show that while models such as Qwen2.5-VL and UI-TARS perform competitively, most MLLMs still exhibit clear weaknesses in reflective decision-making and post-action self-evaluation, limiting their reliability in real-world interactions. We hope GUI-CEval provides a comprehensive and interpretable benchmark to guide capability diagnosis and advance the development of Chinese mobile GUI agents.

GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

Abstract

Recent progress in Multimodal Large Language Models (MLLMs) has enabled mobile GUI agents capable of visual perception, cross-modal reasoning, and interactive control. However, existing benchmarks are largely English-centric and fail to capture the linguistic and interaction characteristics of the Chinese mobile ecosystem. They also focus on isolated skills such as GUI grounding or offline agent, lacking a unified and fine-grained framework to assess the full capability chain from perception to execution. To address this gap, we introduce GUI-CEval, the first comprehensive benchmark for Chinese mobile GUI agents, built entirely on physical device environments. GUI-CEval spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation. All data are collected and verified through multi-stage manual processes to ensure authenticity and reproducibility. Extensive experiments on 20 representative MLLMs and multi-agent systems show that while models such as Qwen2.5-VL and UI-TARS perform competitively, most MLLMs still exhibit clear weaknesses in reflective decision-making and post-action self-evaluation, limiting their reliability in real-world interactions. We hope GUI-CEval provides a comprehensive and interpretable benchmark to guide capability diagnosis and advance the development of Chinese mobile GUI agents.
Paper Structure (30 sections, 8 figures, 2 tables)

This paper contains 30 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agent.
  4. GUI Benchmarks.
  5. GUI-CEval
  6. Task Definition
  7. Foundation Tasks
  8. Perception Task.
  9. Planning Tasks.
  10. Reflection Tasks.
  11. Evaluation Tasks.
  12. Application Tasks
  13. GUI Grounding.
  14. Offline Agent.
  15. Online Agent.
  16. ...and 15 more sections

Figures (8)

  • Figure 1: Results of six representative multimodal large language models across seven tasks defined in GUI-CEval. The uneven radar profiles show that GUI-CEval offers a comprehensive examination of perception–to–execution capabilities and poses a substantial challenge to current models.
  • Figure 2: A representative example illustrating how GUI-CEval provides a comprehensive analysis of a Chinese mobile instruction. It evaluates realistic GUI application ability using tasks constructed from real mobile environments and trajectories, while also assessing atomic skills via single-answer multiple-choice questions, enabling developers to diagnose and improve model weaknesses.
  • Figure 3: The pipeline of GUI-CEval, covering task definition, data collection, human annotation, and multi-stage quality control.
  • Figure 4: Overall statistics of the GUI-CEval, which consists of 4,194 multimodal QA and 4,028 GUI-agent application tasks.
  • Figure 5: Performance trends of representative models across six capability dimensions under varying input resolutions.
  • ...and 3 more figures