WinClick: GUI Grounding with Multimodal Large Language Models

TL;DR

This work tackles GUI grounding for desktop Windows environments by removing reliance on structured data and leveraging visual inputs only. It introduces WinClick, a Windows-focused GUI agent built on a fine-tuned Phi3-vision backbone and augmented with a novel GUI grounding data alignment pipeline, enabling robust forward and reverse grounding tasks expressed as $P(y|S,x)$ and $P(x|S,y)$. To support evaluation, the authors propose WinSpot, a Windows-specific benchmark with over 1,000 images and 5,000 instruction-click pairs across 14 apps, plus comparison against diverse baselines on ScreenSpot. Empirical results show WinClick achieving state-of-the-art performance with an average of $56.1\%$ grounding accuracy (full fine-tune) and substantial gains over the base Phi-3 Vision, validating the importance of GUI-grounding pre-training and data alignment for desktop automation. The work also highlights practical considerations and provides a grounded path toward scalable, cross-desktop GUI automation with a Windows-focused benchmark resource.

Abstract

Graphical User Interface (GUI) tasks are vital for automating workflows such as software testing, user interface navigation. For users, the GUI is the most intuitive platform for interacting with a computer. Previous work identified a key challenge in developing visual GUI agents: GUI grounding - the ability to accurately locate screen elements based on instructions. However, most existing GUI agents rely on structured data formats like DOM or HTML files in training or inferencing, which are inaccessible across all applications, particular in a general desktop environments such as Windows OS. To address this, we introduce WinClick, a novel visual GUI agent developed in Windows platform. WinClick leverages screenshots to detect actionable regions. To overcome the challenge of GUI grounding, we enhance WinClick with GUI grounding pre-training and propose an LLM-based method for aligning GUI grounding data. Additionally, we introduce WinSpot, the first comprehensive benchmark for GUI grounding on Windows. Our experiments demonstrate that WinClick, combined with GUI grounding pre-training, significantly outperforms existing baselines, offering a scalable solution for GUI automation in desktop environments. WinSpot is publicly available at https://github.com/zackhuiiiii/WinSpot.

Figures (7)

• Figure 1: GUI Agent subtask, GUI grounding
• Figure 2: WinClick's full workflow
• Figure 3: Overview of the Data Alignment Process: (a) illustrates the data filtering strategy using the Phi3 vision model, (b) shows the input and output of icon grounding with the in-house ViT-BERT model, and (c) demonstrates the use of LLMs for GUI and description alignment.
• Figure 4: (a) Traditional methods rely on HTML or DOM files to locate icons during data construction. (b) Our proposed data alignment framework requires only raw screenshot images.
• Figure 5: WinSpot examples