FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection

TL;DR

This work tackles inefficiency in UI grounding by addressing extreme visual-token redundancy and positional fragility in high-resolution UI screenshots. It introduces FocusUI, a framework that jointly learns an instruction-aware patch saliency score and applies a PosPad technique to preserve positional continuity during token reduction, enabling substantial speedups with minimal accuracy loss. The method fuses bounding-box overlap with a UI-graph prior to supervise a lightweight saliency scorer, and integrates seamlessly with VLMs like Qwen2.5-VL and Qwen3-VL across multiple sizes. Comprehensive experiments on four UI grounding benchmarks show that FocusUI achieves state-of-the-art accuracy-efficiency trade-offs, outperforming GUI-specific baselines and general pruning methods, with notable gains at 30–50% token retention. The practical impact lies in enabling faster, memory-efficient UI grounding suitable for real-time GUI agents and interactive systems, while pointing to future work on temporal modeling for multi-turn interactions.

Abstract

Vision-Language Models (VLMs) have shown remarkable performance in User Interface (UI) grounding tasks, driven by their ability to process increasingly high-resolution screenshots. However, screenshots are tokenized into thousands of visual tokens (e.g., about 4700 for 2K resolution), incurring significant computational overhead and diluting attention. In contrast, humans typically focus on regions of interest when interacting with UI. In this work, we pioneer the task of efficient UI grounding. Guided by practical analysis of the task's characteristics and challenges, we propose FocusUI, an efficient UI grounding framework that selects patches most relevant to the instruction while preserving positional continuity for precise grounding. FocusUI addresses two key challenges: (1) Eliminating redundant tokens in visual encoding. We construct patch-level supervision by fusing an instruction-conditioned score with a rule-based UI-graph score that down-weights large homogeneous regions to select distinct and instruction-relevant visual tokens. (2) Preserving positional continuity during visual token selection. We find that general visual token pruning methods suffer from severe accuracy degradation on UI grounding tasks due to broken positional information. We introduce a novel PosPad strategy, which compresses each contiguous sequence of dropped visual tokens into a single special marker placed at the sequence's last index to preserve positional continuity. Comprehensive experiments on four grounding benchmarks demonstrate that FocusUI surpasses GUI-specific baselines. On the ScreenSpot-Pro benchmark, FocusUI-7B achieves a performance improvement of 3.7% over GUI-Actor-7B. Even with only 30% visual token retention, FocusUI-7B drops by only 3.2% while achieving up to 1.44x faster inference and 17% lower peak GPU memory.

Figures (8)

• Figure 1: FocusUI is an efficient UI grounding framework that selects instruction-relevant visual tokens while preserving positional continuity. Study 1 provides motivation to address visual redundancy in UI grounding tasks, and Study 2 demonstrates the effectiveness of the our position-preserving selection.
• Figure 2: Overview of our proposed FocusUI. (a) Illustration of how the Instruction-to-Patch saliency score is constructed. (b) Query-guided Saliency Scorer and token selection. (c) Overall UI grounding framework illustrating how PosPad is applied to dropped sequences to preserve positional continuity. For clarity, we omit the system prompt in the token sequence.
• Figure 3: Illustrative example of building the Instruction-to-Patch saliency score. (a) Screenshot $I$ with ground-truth bounding box $b_{gt}$. (b) Bounding-box saliency score $S_{\mathrm{bbox}}$. (c) Union-find results. (d) Size of each connected component $n_u$. (e) UI-graph saliency score $S_{\mathrm{uig}}$. (f) Fused supervision $S_{\mathrm{Ins}2\mathrm{Patch}}$ by combining (d) and (e). Brighter regions represent positive patches and darker regions represent negative patches.
• Figure 4: Illustration of PosPad sequence transformation for positional continuity preservation via an example 2D image (2$\times$3 patches) and its 1D sequence. A learnable <pos_pad> marker is placed at the last index of each contiguous sequence of dropped visual tokens, as illustrated by strategy (d).
• Figure 5: Qualitative visualization of predicted saliency heatmaps and retained patches under a retention ratio$r=\mathbf{30\%}$. Black regions denote dropped visual tokens that are not consumed by the LM during decoding. Examples are taken from the ScreenSpot-V2 and ScreenSpot-Pro benchmarks, spanning web, desktop, and mobile interfaces.