GUIrilla: A Scalable Framework for Automated Desktop UI Exploration

TL;DR

GUIrilla addresses the data scarcity challenge in desktop GUI automation by introducing an automated macOS-focused framework that leverages the native Accessibility API and three GPT-4–based agents to crawl full-desktop environments, construct hierarchical GUI graphs, and generate function-oriented tasks. The project releases GUIrilla-Task, GUIrilla-Gold, GUIrilla-See models, and the macapptree toolkit, enabling reproducible data collection and grounding research. Empirical results show data-efficient, high-fidelity grounding; GUIrilla-See models trained on GUIrilla-Task outperform synthetic baselines on challenging full-desktop tasks and demonstrate cross-OS transfer, despite single-platform training. The work provides a scalable pathway for open desktop autonomy research, while addressing ethical and practical deployment considerations and outlining avenues for extending coverage to more platforms and more advanced agentic exploration.

Abstract

Autonomous agents capable of operating complex graphical user interfaces (GUIs) have the potential to transform desktop automation. While recent advances in large language models (LLMs) have significantly improved UI understanding, navigating full-window, multi-application desktop environments remains a major challenge. Data availability is limited by costly manual annotation, closed-source datasets and surface-level synthetic pipelines. We introduce GUIrilla, an automated scalable framework that systematically explores applications via native accessibility APIs to address the critical data collection challenge in GUI automation. Our framework focuses on macOS - an ecosystem with limited representation in current UI datasets - though many of its components are designed for broader cross-platform applicability. GUIrilla organizes discovered interface elements and crawler actions into hierarchical GUI graphs and employs specialized interaction handlers to achieve comprehensive application coverage. Using the application graphs from GUIrilla crawler, we construct and release GUIrilla-Task, a large-scale dataset of 27,171 functionally grounded tasks across 1,108 macOS applications, each annotated with full-desktop and window-level screenshots, accessibility metadata, and semantic action traces. Empirical results show that tuning LLM-based agents on GUIrilla-Task significantly improves performance on downstream UI tasks, outperforming synthetic baselines on the ScreenSpot Pro benchmark while using 97% less data. We also release macapptree, an open-source library for reproducible collection of structured accessibility metadata, along with the full GUIrilla-Task dataset, the manually verified GUIrilla-Gold benchmark, and the framework code to support open research in desktop autonomy.

Figures (8)

• Figure 1: Parsed hierarchical tree structure from the Session application. Each node represents a UI state, containing the full accessibility tree along with a screenshot of the interface, and the edges denote GUIrilla crawler actions. The hierarchy reflects a sequence of interactions as the agent interacts with application UI, forming the application-specific graph.
• Figure 2: Architecture of the GUIrilla framework. The GUIrilla crawler, equipped with various UI handlers, processes an application bundle using input parameters and installer routines. It interacts with autonomous GPT-4 agents (Input, Order, and Login Agents) to navigate the application. The resulting output graph is refined by a Task Postprocessing Agent (GPT-4), which handles uninstallation and graph cleanup, ultimately producing a structured task list. The dashed line denotes the optional usage of LLMs for app exploration.
• Figure 3: Example sample from our dataset. Left: screenshot of the macOS desktop interface. Upper right: target element cropped. Lower right: a segment of the accessibility tree.
• Figure 4: Number of apps and associated tasks per genre. For each genre, the left bar shows the number of apps, and the right bar shows the number of tasks. Colors distinguish between the two quantities. The figure highlights disparities between app availability and task density across categories.
• Figure 5: Distributions of tasks across element types (left) and task types (right).