WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point

TL;DR

WorldGUI, a benchmark covering ten widely used desktop and web applications with tasks instantiated under diverse, systematically constructed initial states, and WorldGUI-Agent, a simple and model-agnostic framework that organizes planning and execution around three critique stages, improving reliability in dynamic environments are presented.

Abstract

Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environment deviates from a canonical initial state. In real applications, users often invoke assistance mid-workflow, where software may be partially configured, steps may have been executed in different orders, or the interface may differ from its default setup. Such task-state variability is pervasive but insufficiently evaluated in existing GUI benchmarks. To address this gap, we introduce WorldGUI, a benchmark covering ten widely used desktop and web applications with tasks instantiated under diverse, systematically constructed initial states. These variations capture realistic human-computer interaction settings and enable diagnostic evaluation of an agent's ability to recover, adapt plans, and handle non-default contexts. We further present WorldGUI-Agent, a simple and model-agnostic framework that organizes planning and execution around three critique stages, improving reliability in dynamic environments. Experiments demonstrate that state-of-the-art GUI agents exhibit substantial performance degradation under non-default initial conditions, revealing limited robustness and fragile planning behaviors. Our benchmark and framework provide a foundation for developing more adaptable and reliable GUI agents. The code and data are available at https://github.com/showlab/WorldGUI.

Figures (25)

• Figure 1: WorldGUI. Left: WorldGUI creates pre-actions for each meta task, leading to different initial states. It successfully reflects the real-world human-computer interaction process. Right: components in WorldGUI.
• Figure 2: The left shows software taxonomy in WorldGUI. The right shows the distribution of task length.
• Figure 3: The distribution of different augmentation types.
• Figure 4: The distribution of different task difficulty.
• Figure 5: An example of augmenting one GUI task with manually aug the initial state and then using the execution scripts and corresponding agents to obtain the pre-action for each augmented case.