ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks

TL;DR

This work addresses the gap between offline static and online dynamic evaluation for mobile GUI agents by introducing ColorBench, a graph-structured benchmark that staticly simulates dynamic interactions. It defines a strongly connected graph where nodes represent screen states and edges actions, enabling multi-path, atomic-task evaluation across 175 tasks and 15 atomic capabilities. Through extensive experiments on open- and closed-source models, ColorBench reveals grounding, planning, memory, and decomposition limitations, and demonstrates that graph-structured evaluation provides stable, reproducible yet richly interactive assessment. The approach offers practical pathways to improve agents for complex, long-horizon mobile tasks and highlights where to focus future model improvements.

Abstract

The rapid advancement of multimodal large language models has enabled agents to operate mobile devices by directly interacting with graphical user interfaces, opening new possibilities for mobile automation. However, real-world mobile tasks are often complex and allow for multiple valid solutions. This contradicts current mobile agent evaluation standards: offline static benchmarks can only validate a single predefined "golden path", while online dynamic testing is constrained by the complexity and non-reproducibility of real devices, making both approaches inadequate for comprehensively assessing agent capabilities. To bridge the gap between offline and online evaluation and enhance testing stability, this paper introduces a novel graph-structured benchmarking framework. By modeling the finite states observed during real-device interactions, it achieves static simulation of dynamic behaviors. Building on this, we develop ColorBench, a benchmark focused on complex long-horizon tasks. It supports evaluation of multiple valid solutions, subtask completion rate statistics, and atomic-level capability analysis. ColorBench contains 175 tasks (74 single-app, 101 cross-app) with an average length of over 13 steps. Each task includes at least two correct paths and several typical error paths, enabling quasi-dynamic interaction. By evaluating ColorBench across various baselines, we discover limitations of existing models and propose improvement directions and feasible technical pathways to enhance agents' performance on complex, long-horizon problems based on experimental results. Code and data are available at: https://github.com/MadeAgents/ColorBench.

Figures (5)

• Figure 1: Structure of the ColorBench. It illustrates multi-path solutions, reflective backtracking and automated evaluation milestones, demonstrating how the graph is structured and utilized. Each node represents a screen list.
• Figure 2: Advantage of graph-structured benchmark. Our constructed ColorBench possesses these advantages. The upper-left corner demonstrates multiple solutions to one problem, supporting diverse execution paths for the same task. The lower-left corner illustrates how the graph's strongly connected structure effectively enables model collaboration capabilities such as backtracking and reflection. The upper-right corner showcases the graph's inherent suitability for complex long-horizon tasks and its support for atomic capability evaluation. The lower-right corner highlights the graph's superior controllability and reproducibility compared to dynamic evaluation.
• Figure 3: ColorBench statistics. The results do not include actions "navigate back", "navigate home", or "open app".
• Figure 4: Statistics on the number of actions that all experienced models at each node in the graph tend to perform, do not include "navigate back", "navigate home" and "open app" which can be executed on any interface.
• Figure 5: System prompt templates for merging graph. In practical use, given the language of the application, we employed Chinese-language prompts.