Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation

Dezhi Ran, Mengzhou Wu, Hao Yu, Yuetong Li, Jun Ren, Yuan Cao, Xia Zeng, Haochuan Lu, Zexin Xu, Mengqian Xu, Ting Su, Liangchao Yao, Ting Xiong, Wei Yang, Yuetang Deng, Assaf Marron, David Harel, Tao Xie

TL;DR

This paper presents Sphinx, a multi-dimensional benchmark to evaluate foundation models on goal-based mobile UI navigation in industrial settings. It introduces a five-capability evaluation framework (goal understanding, app knowledge, planning, grounding, and instruction following) and a suite of toolkits and tasks, including WeChat QA tests and 100 popular apps, to diagnose specific failure modes. Empirical results across eight models show low end-to-end effectiveness, with significant disparities between text- and vision-based models, revealing critical UI-specific bottlenecks in grounding and planning. The study also analyzes how model deficiencies constrain UI navigation agents, yielding seven actionable lessons and directing future work toward UI-focused fine-tuning and auxiliary support to close the gap to real-world deployment.

Abstract

Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation.

Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation

TL;DR

This paper presents Sphinx, a multi-dimensional benchmark to evaluate foundation models on goal-based mobile UI navigation in industrial settings. It introduces a five-capability evaluation framework (goal understanding, app knowledge, planning, grounding, and instruction following) and a suite of toolkits and tasks, including WeChat QA tests and 100 popular apps, to diagnose specific failure modes. Empirical results across eight models show low end-to-end effectiveness, with significant disparities between text- and vision-based models, revealing critical UI-specific bottlenecks in grounding and planning. The study also analyzes how model deficiencies constrain UI navigation agents, yielding seven actionable lessons and directing future work toward UI-focused fine-tuning and auxiliary support to close the gap to real-world deployment.

Abstract

Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation.
Paper Structure (35 sections, 2 figures, 9 tables)

This paper contains 35 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. UI Navigation Benchmarks
  4. Foundation Models for UI Navigation
  5. Sphinx Benchmark
  6. Overview of Sphinx
  7. Toolkits for Multi-dimensional Evaluation
  8. Trajectory-based Evaluator
  9. Knowledge Probing
  10. Completion Judgment
  11. Low-level Instruction
  12. Invariants
  13. Task Collection
  14. Goal-based UI Navigation Task.
  15. Knowledge Probing Tasks.
  16. ...and 20 more sections

Figures (2)

  • Figure 1: An example of goal-based mobile UI navigation on WeChat with a foundation model.
  • Figure 2: Visualization of popular FMs' five key capabilities required for mobile UI navigation and end-to-end effectiveness on Sphinx. UI-specific capabilities are the bottlenecks for FM-based mobile UI navigation.