AUTO-Explorer: Automated Data Collection for GUI Agent

TL;DR

The paper tackles GUI data scarcity for training agents that interpret natural language commands by introducing Auto-Explorer, an autonomous GUI data-collection framework with a GUI parser and an exploration module. It also introduces the UIXplore benchmark to evaluate exploration quality and a 4,800-sample GUI-element grounding test set to assess grounding performance. Through extensive experiments, Auto-Explorer outperforms baselines and SOTA in both software and web environments, enabling faster and more robust fine-tuning of multimodal models such as Qwen2-VL-2B. The work demonstrates that automated, minimally annotated exploration can significantly improve GUI understanding and grounding, with strong implications for adaptable GUI agents across diverse software.

Abstract

Recent advancements in GUI agents have significantly expanded their ability to interpret natural language commands to manage software interfaces. However, acquiring GUI data remains a significant challenge. Existing methods often involve designing automated agents that browse URLs from the Common Crawl, using webpage HTML to collect screenshots and corresponding annotations, including the names and bounding boxes of UI elements. However, this method is difficult to apply to desktop software or some newly launched websites not included in the Common Crawl. While we expect the model to possess strong generalization capabilities to handle this, it is still crucial for personalized scenarios that require rapid and perfect adaptation to new software or websites. To address this, we propose an automated data collection method with minimal annotation costs, named Auto-Explorer. It incorporates a simple yet effective exploration mechanism that autonomously parses and explores GUI environments, gathering data efficiently. Additionally, to assess the quality of exploration, we have developed the UIXplore benchmark. This benchmark creates environments for explorer agents to discover and save software states. Using the data gathered, we fine-tune a multimodal large language model (MLLM) and establish a GUI element grounding testing set to evaluate the effectiveness of the exploration strategies. Our experiments demonstrate the superior performance of Auto-Explorer, showing that our method can quickly enhance the capabilities of an MLLM in explored software.

Figures (6)

• Figure 1: Auto-Explorer is a data collection agent capable of autonomous exploration within a given environment. It utilizes various tools such as UI Automation (UIA), Optical Character Recognition (OCR), and icon template matching algorithms to parse the content of images. The agent selects UI elements to interact with from these parsed elements, continuously exploring new environmental states.
• Figure 2: This diagram categorizes software and websites used in our benchmark, with the outer ring's color coding indicating usage: light brown for software or websites used in both exploration environments and GUI grounding testing set, purple for exploration-only ones, and uncolored for the testing set.
• Figure 3: Pipeline of Auto-Explorer: Auto-Explorer consists of two primary components: 1) GUI parser, which automatically parses UI elements from screenshots, and 2) Explore Module, tasked with determining subsequent actions to uncover new environment states. Initially, the model selects a random unclicked button. After each action, the Difference Spot Module checks if there are new elements in the UI, and then the Critic Module will choose a random action from these for execution. If no new elements emerge post-action, exploration stops.
• Figure 4: Accuracy Performance (%) on UIXplore against unique actions rate.
• Figure 5: Impact of Fine-Tuning Sample Size on Model Accuracy: This graph displays the progression of accuracy improvements for All Accuracy, Text Accuracy, Icon Accuracy. The gray line represents the results of OmniParser.