GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration

TL;DR

The paper tackles the problem that GUI grounding models trained on broad datasets underperform in novel environments. It introduces GUI-Bee, an MLLM-based autonomous explorer, and a training-free Q-ICRL mechanism to efficiently collect environment-specific data and continuously fine-tune grounding models. The NovelScreenSpot benchmark evaluates alignment of multiple GUI grounding models to five unseen environments, highlighting improvements in action-grounding and action-outcome queries, especially with Vision+A11y inputs. Ablation studies show Q-ICRL boosts exploration efficiency, and environment-specific data yields substantial gains over baselines, enabling practical deployment of environment-aware GUI agents. Overall, the work provides a concrete pipeline for adapting GUI grounding to new environments and demonstrates the value of targeted data collection and continual fine-tuning.

Abstract

Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee. Project page: https://gui-bee.github.io

Figures (15)

• Figure 1: Our environment-aligned GUI action grounding model, based on GUI grounding models from prior works, is aligned to novel environments. Our proposed alignment process includes first exploring the specific novel environment with the GUI-Bee agent to generate the exploration graph and then fine-tuning the model with the data from the exploration graph. In the inference example at the bottom, the models encounter a query requiring knowledge of an environment-specific action outcome, which highlights the importance of the proposed alignment process.
• Figure 2: An example of the exploration graph showing screens connected by actions. Middle: A zoomed-in view of the graph with examples of $i^t$ and $i^{t+1}$ and some explored and unexplored actions (ever/never selected during the exploration).
• Figure 3: Example of predicting the $\hat{Q}(a^t_x)$ with the MLLM through in-context learning (ICL). Two example actions $(a_{\text{eg1}}, a_{\text{eg2}})$ marked by bounding boxes 2 and 3 are provided as the context along with their Q values.
• Figure 4: Overall average model performance improvements (P.I.) in the NovelScreenSpot benchmark, compared with the average model P.I. on queries related to action outcome. The consistent P.I. between these two categories shows the proposed alignment improves model performance evenly.
• Figure 5: Mean and standard deviation of Depth-fixed DOM Diversity Counts (D3C) at various exploration steps across three runs in three environments. GUI-Bee agent demonstrates a wider exploration coverage.