GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents

TL;DR

GUI-360° addresses the scarcity of real-world desktop task data by delivering a large-scale, automated pipeline that yields over 1.2 million executed action steps across Word, Excel, and PowerPoint, with full multimodal annotations and a hybrid GUI+API action space. It provides three canonical tasks—GUI grounding, screen parsing, and action prediction—within a unified benchmark and accompanying GUI-360°-Bench, enabling robust evaluation and training of computer-using agents. The framework combines Prototypical Query Sourcing, environment templates, automated task instantiation, and a two-stage execution strategy (GPT-4o then GPT-4.1) to achieve scalable data collection, validated by an LLM-based judge and structured post-processing. Across grounding, parsing, and action-prediction tasks, state-of-the-art models show clear limitations without adaptation, but supervised fine-tuning and RL on GUI-360° yield substantial gains, illustrating the dataset’s value for advancing reliable, cross-application desktop automation. The work culminates in public release of GUI-360°, the benchmark, and the data collection framework, aiming to catalyze research on robust, real-world desktop CUAs with broad practical impact.

Abstract

We introduce GUI-360$^\circ$, a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs). CUAs present unique challenges and is constrained by three persistent gaps: a scarcity of real-world CUA tasks, the lack of automated collection-and-annotation pipelines for multi-modal trajectories, and the absence of a unified benchmark that jointly evaluates GUI grounding, screen parsing, and action prediction. GUI-360$^\circ$ addresses these gaps with an LLM-augmented, largely automated pipeline for query sourcing, environment-template construction, task instantiation, batched execution, and LLM-driven quality filtering. The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications, and includes full-resolution screenshots, accessibility metadata when available, instantiated goals, intermediate reasoning traces, and both successful and failed action trajectories. The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space that reflects modern agent designs. Benchmarking state-of-the-art vision--language models on GUI-360$^\circ$ reveals substantial out-of-the-box shortcomings in grounding and action prediction; supervised fine-tuning and reinforcement learning yield significant gains but do not close the gap to human-level reliability. We release GUI-360$^\circ$ and accompanying code to facilitate reproducible research and accelerate progress on robust desktop CUAs. The full dataset has been made public on https://huggingface.co/datasets/vyokky/GUI-360.

Figures (4)

• Figure 1: An illustration of GUI Grounding, Screen Parsing and Action Prediction tasks included in our GUI-360$^\circ$.
• Figure 2: The data collection pipeline for GUI-360$^\circ$.
• Figure 3: The overall architecture of the TrajAgent.
• Figure 4: Dataset Composition. For each app, tasks are divided into six categories according to their core operational intent.