GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents

TL;DR

GUI-Eyes tackles adaptive perception for GUI grounding by introducing a tool-augmented two-stage inference framework. It defines a unified, spatially informed reward $R(\tau)=\lambda_{acc} R_{acc}+\lambda_{format} R_{format}+\lambda_{tool} R_{tool}$, where $R_{tool}$ combines Center Proximity and Region Overlap to guide tool usage, and supports Crop and Zoom tools. The method executes Stage 1 Active Perception Planning and Stage 2 Focused Reasoning, coordinated by a GRPO-based end-to-end policy. On ScreenSpot-Pro, GUI-Eyes-3B achieves 44.8% grounding accuracy with only 3k labeled samples, outperforming both supervised and RL baselines and demonstrating strong data efficiency and cross-platform generalization across Android, Desktop, and Web tasks.

Abstract

Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.

Figures (8)

• Figure 1: Performance Scaling of Multimodal UI Understanding Models on the ScreenSpot-Pro Benchmark. Our method achieves state-of-the-art performance.
• Figure 1: An Example of Active Perception with Cropping. Offset maps the point from crop to original image.
• Figure 2: Overview of the GUI-Eyes Framework. The top illustrates a rollout example with optional visual tool invocation, together with a tool-specific reward function that combines spatial proximity and region overlap relative to the ground-truth. The bottom depicts the progressive inference architecture and end-to-end training pipeline, where the two-stage decision process is guided by stage-specific prompts, and visual inputs are dynamically generated through previously applied visual tools.
• Figure 2: An Example of Direct Grounding without Tool.
• Figure 3: Inference Example of Tool-Augmented Reasoning with Cropping in a GUI Task.