Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen, Shuxian Li, Kaer Huang, Yanzhe Jing, Yiqiang Yan, Bo Zhang, Chenghao Jiang, Borui Zhang, Jiwen Lu

Abstract

GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

Abstract

GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.
Paper Structure (11 sections, 4 equations, 1 figure, 2 tables)

This paper contains 11 sections, 4 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Instruction Refinement
  4. Conditional Zoom-in Grounding
  5. Dataset Construction
  6. Training Details
  7. Conditional Zoom-In Strategy
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Conclusion

Figures (1)

  • Figure 1: Overall grounding inference pipeline. Given a GUI screenshot and a natural language instruction, a refinement model first rewrites the instruction into a direct and detailed command. The grounding model then takes the rewritten instruction and the screenshot as input and outputs both the click-point and the bounding box of the target element. If the predicted bounding box is larger than a certain threshold, the click-point is directly used as the final output. Otherwise, the screenshot is zoomed in around the click-point and fed into the grounding model again for a second round of inference.