GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent

TL;DR

GUI-explorer presents a training-free GUI agent that tackles UI misinterpretation and knowledge obsolescence by marrying Function-aware Trajectory exploration with Transition-aware Knowledge mining. It generates anchor-guided exploration goals, autonomously collects diverse interaction trajectories, and unsupervisedly extracts atomic screen-operation logic to form a dynamic knowledge vector store. A visual-semantic retrieval and a learned Knowledge Ranker fuse this knowledge with real-time observations to produce precise, state-consistent guidance, achieving SOTA results on SPA-Bench and AndroidWorld. The work also introduces GUI-KRB, a benchmark exposing prior knowledge gaps and dynamic reasoning limitations in current MLLMs, and demonstrates robustness and cross-domain generalization, while acknowledging limitations and outlining future work toward web/desktop extension and efficiency improvements.

Abstract

GUI automation faces critical challenges in dynamic environments. MLLMs suffer from two key issues: misinterpreting UI components and outdated knowledge. Traditional fine-tuning methods are costly for app-specific knowledge updates. We propose GUI-explorer, a training-free GUI agent that incorporates two fundamental mechanisms: (1) Autonomous Exploration of Function-aware Trajectory. To comprehensively cover all application functionalities, we design a Function-aware Task Goal Generator that automatically constructs exploration goals by analyzing GUI structural information (e.g., screenshots and activity hierarchies). This enables systematic exploration to collect diverse trajectories. (2) Unsupervised Mining of Transition-aware Knowledge. To establish precise screen-operation logic, we develop a Transition-aware Knowledge Extractor that extracts effective screen-operation logic through unsupervised analysis the state transition of structured interaction triples (observation, action, outcome). This eliminates the need for human involvement in knowledge extraction. With a task success rate of 53.7% on SPA-Bench and 47.4% on AndroidWorld, GUI-explorer shows significant improvements over SOTA agents. It requires no parameter updates for new apps. GUI-explorer is open-sourced and publicly available at https://github.com/JiuTian-VL/GUI-explorer.

Figures (7)

• Figure 1: Comparison of GPT-4o and an user's interpretation of a UI element in QQ Music. The red-bounded icon in the screenshot represents the music recognition feature, but GPT-4o misidentified it. This highlights the challenge of accurately interpreting UI elements in an ecosystem of diverse apps with distinct designs.
• Figure 2: Overview of GUI-explorer. (a) Automatically constructing function-aware exploration goals by analyzing structural information from the GUI environment, followed by systematic exploration to collect diverse function-aware trajectories. (b) Extracting effective screen-operation logic through unsupervised analysis of structured interaction triples (observation, action, outcome), enabling unsupervised knowledge extraction. (c) Performing visual-semantic retrieval between screen visuals and the knowledge vector store to construct Dynamic Guidance achieves dual objectives: preventing UI misinterpretation and ensuring action proposals align with actual UI states.
• Figure 3: Without transition-aware knowledge as reliable prior information, MLLMs may fail to reason correctly due to outdated prior knowledge or diverse GUI designs.
• Figure 4: Ablation study of operational guidance configurations on SPA-Bench: (1) Baseline without dynamic guidance, (2) Guidance derived from cross-environment exploration (AndroidWorld), (3) Guidance generated through in-environment exploration (SPA-Bench).
• Figure 5: Distribution of apps in GUI-KRB.