UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning

TL;DR

This paper introduces the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning, and proposes a two-stage training framework: supervised fine-tuning on synthesized, diverse instructions to instill multi-perspective reasoning.

Abstract

GUI grounding, which maps natural-language instructions to actionable UI elements, is a core capability of GUI agents. Prior works largely treats instructions as a static proxy for user intent, overlooking the impact of instruction diversity and quality on grounding performance. Through a careful investigation of existing grounding datasets, we find a 23.3% flaw rate in their instructions and show that inference-time exploitation of instruction diversity yields up to a substantial 76% relative performance improvement. In this paper, we introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning (SFT) on synthesized, diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning (RL) to optimize pathway selection and composition. Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model demonstrates strong agentic potential, achieving a 74.1% success rate on AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals additional insights such as how reasoning can be formulated to enhance rather than hinder grounding performance, and how our method mitigates policy collapse in the SFT+RL framework. All code and model checkpoints will be publicly released in https://github.com/alibaba/UI-Ins.

Figures (12)

• Figure 1: Performance comparisons of UI-Ins and other state-of-the-art methods.
• Figure 2: Preliminary analysis of GUI Grounding Instructions. (a) Instruction diversity influences performance significantly. (b) Instruction quality problems in existing open-source datasets. (c) Low instruction quality undermines training efficacy.
• Figure 3: Effective instruction perspectives in different GUI scenarios. Samples are from OS-Atlas Dataset and the ground truth bounding box is labeled with a green box beside a yellow star.
• Figure 4: Instruction quality flaws in grounding datasets. Left: Ambiguous match, an instruction maps to multi UI elements. Right: Mismatch, no valid UI element matches the instruction.
• Figure 5: Overview of our high-quality data processing pipeline. The pipeline first preprocesses the ground-truth bounding box, then leverages GPT-4.1 to generate instructions from diverse perspectives, and finally employs a verification stage to filter the results by ensuring a precise alignment between the instruction and the ground truth box.