Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and MCP-Augmented Environments

Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, Yue Wang

TL;DR

MobileWorld tackles saturation and realism gaps in mobile GUI benchmarks by introducing a substantially harder, reproducible benchmark with 201 tasks across 20 apps. It adds novel agent-user interaction and MCP-augmented task categories and implements a planner-executor framework that unifies GUI actions, clarifying dialogues, and external tool calls in a closed loop. The study reveals large performance gaps for current models, especially on agent-user interaction and MCP tasks, and identifies five key research challenges—ambiguity handling, context management for MCP, long-term memory, complex reasoning, and temporal-spatial awareness. The work includes a fully containerized, deterministic evaluation setup with open-source backends and snapshot-based state management, offering a rigorous testbed for next-generation mobile agents. Overall, MobileWorld provides a concrete roadmap for advancing autonomous mobile intelligence toward real-world, multi-domain automation.

Abstract

Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its saturation and motivate the need for a more challenging benchmark. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. To bridge this gap, we introduce MobileWorld, a substantially more challenging benchmark designed to better reflect real-world mobile usage, comprising 201 tasks across 20 applications, while maintaining the same level of reproducible evaluation as AndroidWorld. The difficulty of MobileWorld is twofold. First, it emphasizes long-horizon tasks with cross-application interactions: MobileWorld requires nearly twice as many task-completion steps on average (27.8 vs. 14.3) and includes far more multi-application tasks (62.2% vs. 9.5%) compared to AndroidWorld. Second, MobileWorld extends beyond standard GUI manipulation by introducing novel task categories, including agent-user interaction and MCP-augmented tasks. To ensure robust evaluation, we provide snapshot-based container environment and precise functional verifications, including backend database inspection and task callback APIs. We further develop a planner-executor agentic framework with extended action spaces to support user interactions and MCP calls. Our results reveal a sharp performance drop compared to AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7% and 20.9% success rates, respectively. Our analysis shows that current models struggle significantly with user interaction and MCP calls, offering a strategic roadmap toward more robust, next-generation mobile intelligence.

MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive, and MCP-Augmented Environments

TL;DR

MobileWorld tackles saturation and realism gaps in mobile GUI benchmarks by introducing a substantially harder, reproducible benchmark with 201 tasks across 20 apps. It adds novel agent-user interaction and MCP-augmented task categories and implements a planner-executor framework that unifies GUI actions, clarifying dialogues, and external tool calls in a closed loop. The study reveals large performance gaps for current models, especially on agent-user interaction and MCP tasks, and identifies five key research challenges—ambiguity handling, context management for MCP, long-term memory, complex reasoning, and temporal-spatial awareness. The work includes a fully containerized, deterministic evaluation setup with open-source backends and snapshot-based state management, offering a rigorous testbed for next-generation mobile agents. Overall, MobileWorld provides a concrete roadmap for advancing autonomous mobile intelligence toward real-world, multi-domain automation.

Abstract

Among existing online mobile-use benchmarks, AndroidWorld has emerged as the dominant benchmark due to its reproducible environment and deterministic evaluation; however, recent agents achieving over 90% success rates indicate its saturation and motivate the need for a more challenging benchmark. In addition, its environment lacks key application categories, such as e-commerce and enterprise communication, and does not reflect realistic mobile-use scenarios characterized by vague user instructions and hybrid tool usage. To bridge this gap, we introduce MobileWorld, a substantially more challenging benchmark designed to better reflect real-world mobile usage, comprising 201 tasks across 20 applications, while maintaining the same level of reproducible evaluation as AndroidWorld. The difficulty of MobileWorld is twofold. First, it emphasizes long-horizon tasks with cross-application interactions: MobileWorld requires nearly twice as many task-completion steps on average (27.8 vs. 14.3) and includes far more multi-application tasks (62.2% vs. 9.5%) compared to AndroidWorld. Second, MobileWorld extends beyond standard GUI manipulation by introducing novel task categories, including agent-user interaction and MCP-augmented tasks. To ensure robust evaluation, we provide snapshot-based container environment and precise functional verifications, including backend database inspection and task callback APIs. We further develop a planner-executor agentic framework with extended action spaces to support user interactions and MCP calls. Our results reveal a sharp performance drop compared to AndroidWorld, with the best agentic framework and end-to-end model achieving 51.7% and 20.9% success rates, respectively. Our analysis shows that current models struggle significantly with user interaction and MCP calls, offering a strategic roadmap toward more robust, next-generation mobile intelligence.
Paper Structure (52 sections, 7 equations, 12 figures, 9 tables)

This paper contains 52 sections, 7 equations, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. GUI Agent Benchmarks
  4. Agent-User Interaction and MCP-Augmented Benchmarks
  5. MobileWorld
  6. Task Definition
  7. GUI-Only Tasks
  8. Agent-user Interaction Tasks
  9. MCP-augmented Tasks
  10. Environment Development
  11. Action Space
  12. Stable and Reproducible Environment
  13. Task Evaluation
  14. Benchmark Construction
  15. Task Instructions and Scenarios
  16. ...and 37 more sections

Figures (12)

  • Figure 1: Compared to AndroidWorld, MobileWorld exhibits lower SOTA success rates, longer task horizons, more cross-application tasks, and sharp performance drops for recent models.
  • Figure 2: Beyond traditional GUI-only tasks, MobileWorld includes agent–user interaction tasks and MCP-augmented tasks, each with distinct deterministic evaluation strategies. Left: An example of an agent–user interaction task, in which the agent must proactively request clarification from a simulated user when encountering incomplete information. A GPT-4.1–based simulated user agent is then triggered to provide the requested information, which is embedded in its system prompt. Task completion is verified through the application's callback cache. Right: An example of an MCP-augmented task, where the agent is initialized with a list of GitHub MCP tools and selects the appropriate tool to retrieve README content from a GitHub repository before completing the task via GUI operations. Task completion is verified through backend database inspection.
  • Figure 3: The system architecture of MobileWorld consists of two main components. Left: the host machine is where GUI agents receive task instructions and optionally interact with users for clarification, then choose between GUI actions or MCP tool calls to complete tasks. Right: the docker environment contains an isolated Android ecosystem with emulators, self-hosted app backends, and an evaluator that verifies task completion through text matching, backend database, local storage, and app callbacks.
  • Figure 4: Scenario Distribution. The benchmark predominantly features third-party applications (95%), with system apps comprising the remaining 5% of tasks.
  • Figure 5: Comparison of completion steps between AndroidWorld and MobileWorld.
  • ...and 7 more figures