GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

TL;DR

GUI-AIMA addresses GUI grounding by reframing the task as a coordinate-free visual localization that leverages the intrinsic multimodal attention of large language models. It introduces an anchored attention mechanism using a learnable <ANCHOR> token and visual-sink query tokens to weigh attention heads, enabling patch-wise supervision without extra grounding modules and achieving high data efficiency with only ~85k training images. The method supports a two-step zoom-in at inference to mitigate offset errors on high-resolution screens, and experiments show state-of-the-art performance among 3B-scale models on multiple GUI benchmarks. Overall, GUI-AIMA provides a practical, data-efficient pathway toward accurate, generalizable GUI grounding with strong implications for intelligent agents and user-interface automation.

Abstract

Graphical user interface (GUI) grounding is a key function of computer-use agents, which maps natural-language instructions to actionable screen regions. Existing approaches based on Multimodal Large Language Models (MLLMs) typically formulate it as a text-based coordinate generation task, yet directly generating precise coordinates from visual inputs remains challenging and computationally intensive. An intuitive way to implement GUI grounding is to first select visual patches relevant to the instructions and then determine the precise click location within those patches. Based on the observations that general MLLMs have some native grounding capability, nested within their attentions, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 85k screenshots, demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 59.6% on ScreenSpot-Pro, 63.8% on OSWorld-G and 91.5% on ScreenSpot-v2. Project page: https://github.com/sjz5202/GUI-AIMA

Figures (9)

• Figure 1: An example of GUI-AIMA with optional two-step GUI grounding for high-res screenshots.
• Figure 2: With user query $\mathcal{Q}$, screenshot patches $\mathcal{V}$ and multi-head attentions $\{\mathbf{A}^{l,h}\}_{l\in[L],h\in[H]}$ from the MLLM, the vanilla attention grounding needs additional aggregation between all query tokens' grounding vectors. In our proposed simplified version, a special $\texttt{<ANCHOR>}$ token can learn to implicitly aggregate all query tokens. Then we aggregate grounding vectors of $\texttt{<ANCHOR>}$ token across layers and heads with carefully designed weights to produce the patch-wise predictions.
• Figure 3: Details about how to compute the final patch-wise prediction based on the grounding vectors of the $\texttt{<ANCHOR>}$ token: GUI-AIMA first specifies visual-sink query tokens via computing hidden state similarities between query tokens and visual patches; Then it computes weights $\boldsymbol{w}$ of each attention head based on visual-sink query tokens in \ref{['eq:head_weighting']}; Finally, GUI-AIMA aggregates the grounding vectors of $\texttt{<ANCHOR>}$ token across layers and heads in \ref{['eq:aggre_final']}.
• Figure 4: Analysis experiment results on ScreenSpot-pro. Relax@k measures how many previously incorrect offset predictions are recovered when the ground-truth bounding box is expanded by k visual patches along each dimension. Recovered refers to the number of offset predictions corrected by the second step, while Lost refers to the number of predictions that degrade after the second step.
• Figure 5: Model convergence of the 45k ablation dataset on ScreenSpot-Pro benchmark.