ShowUI-Aloha: Human-Taught GUI Agent

TL;DR

ShowUI-Aloha tackles the scarcity of scalable, high-quality GUI-task training data by introducing a demonstration-driven pipeline that records human desktop interactions and converts them into structured, semantically grounded traces. The system combines a lightweight Recorder, a Learner that builds semantic traces via a Trace Generator, and a Planner–Actor–Executor stack that reason about tasks and execute actions on real desktops, with temporal memory to handle long-horizon workflows. Evaluated on 361 OSWorld-style tasks across macOS and Windows, ShowUI-Aloha achieves a robust 60.1% end-to-end success and outperforms unguided GUI models, while ablations confirm the importance of both human-taught traces and memory-enabled planning. The work provides a practical, open-source foundation for demonstration-driven GUI agents capable of adapting to layout drift and real-world variability, with potential impact on automated software use and productivity tooling.

Abstract

Graphical User Interfaces (GUIs) are central to human-computer interaction, yet automating complex GUI tasks remains a major challenge for autonomous agents, largely due to a lack of scalable, high-quality training data. While recordings of human demonstrations offer a rich data source, they are typically long, unstructured, and lack annotations, making them difficult for agents to learn from.To address this, we introduce ShowUI-Aloha, a comprehensive pipeline that transforms unstructured, in-the-wild human screen recordings from desktop environments into structured, actionable tasks. Our framework includes four key components: A recorder that captures screen video along with precise user interactions like mouse clicks, keystrokes, and scrolls. A learner that semantically interprets these raw interactions and the surrounding visual context, translating them into descriptive natural language captions. A planner that reads the parsed demonstrations, maintains task states, and dynamically formulates the next high-level action plan based on contextual reasoning. An executor that faithfully carries out these action plans at the OS level, performing precise clicks, drags, text inputs, and window operations with safety checks and real-time feedback. Together, these components provide a scalable solution for collecting and parsing real-world human data, demonstrating a viable path toward building general-purpose GUI agents that can learn effectively from simply observing humans.

Figures (16)

• Figure 1: Overview and evaluation of ShowUI-Aloha.Left: Human-taught demonstrations are converted into grounded action traces, which are lifted into trace- and prompt-guided plans and executed on real desktop environments. Middle: Qualitative comparisons across representative multi-step desktop tasks show that Aloha avoids common failure modes of unguided agents, such as context drift, unsupported actions, and stuck states. Right: Quantitative comparison on 361 OSWorld-style tasks executed on Windows and macOS demonstrates that human-guided planning enables higher end-to-end task success than existing autonomous and agentic baselines.
• Figure 2: Overview of the Aloha paradigm for GUI agents. Instead of relying on trial-and-error interaction, Aloha leverages a single human demonstration to distill reusable task guidance, which is then consistently applied to new task variants and interface layouts, enabling stable and generalizable execution across changing interfaces.
• Figure 3: Overview of the Aloha workflow. Human demonstrations are recorded and converted into structured action traces. The actor uses the task prompt and screenshots to generate an execution plan, while the executor performs each action on the computer.
• Figure 4: User-facing interface of the ShowUI-Aloha Recorder. The recorder presents a minimal floating control panel (top right) for starting and stopping captures, while a modal dialog allows users to name or rename each recording with clear constraints on valid characters. These utilities support organized, large-scale data collection and facilitate downstream processing.
• Figure 5: Raw screen frames captured by the Aloha Recorder. These consecutive frames illustrate the natural visual trajectory present in human desktop demonstrations. The Recorder captures full-resolution frames at high frequency, preserving the fine-grained cursor dynamics, UI transitions, and subtle motion patterns that are essential for downstream action cleaning and trace generation.