HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration

TL;DR

HyperClick tackles overconfidence in GUI grounding by coupling spatial grounding with calibrated uncertainty. It introduces a truncated Gaussian confidence model over UI elements, adaptive variance, and a dual reward framework (binary correctness and confidence calibration) optimized via Group Relative Policy Optimization. Empirical results across seven challenging benchmarks show state-of-the-art accuracy and well-calibrated confidence, enabling introspective self-criticism and safer GUI automation. The approach offers practical gains in reliability for autonomous GUI agents and suggests extensions to planning in multimodal agentic systems.

Abstract

Autonomous Graphical User Interface (GUI) agents rely on accurate GUI grounding, which maps language instructions to on-screen coordinates, to execute user commands. However, current models, whether trained via supervised fine-tuning (SFT) or reinforcement fine-tuning (RFT), lack self-awareness of their capability boundaries, leading to overconfidence and unreliable predictions. We first systematically evaluate probabilistic and verbalized confidence in general and GUI-specific models, revealing a misalignment between confidence and actual accuracy, which is particularly critical in dynamic GUI automation tasks, where single errors can cause task failure. To address this, we propose HyperClick, a novel framework that enhances reliable GUI grounding through uncertainty calibration. HyperClick introduces a dual reward mechanism, combining a binary reward for correct actions with a truncated Gaussian-based spatial confidence modeling, calibrated using the Brier score. This approach jointly optimizes grounding accuracy and confidence reliability, fostering introspective self-criticism. Extensive experiments on seven challenge benchmarks show that HyperClick achieves state-of-the-art performance while providing well-calibrated confidence. By enabling explicit confidence calibration and introspective self-criticism, HyperClick reduces overconfidence and supports more reliable GUI automation.

Figures (3)

• Figure 1: Overview of accuracy and confidence evaluation on ScreenSpot-Pro. (a): Illustration of probabilistic and verbalized confidence. Probabilistic confidence represents the probability of the model generating the next token corresponding to the target coordinates, while verbalized confidence indicates the model’s self-reported certainty about its output in natural language. (b): Comparisons of accuracy, probabilistic confidence, and verbalized confidence for several general-purpose and GUI-specific models on the ScreenSpot-Pro benchmark. The models exhibit a higher confidence in their answers than in the accuracy that they actually achieve.
• Figure 2: Framework of the proposed HyperClick, optimized with Group Relative Policy Optimization (GRPO). Given a screenshot and an instruction, the policy generates $N$ predictions, which are evaluated by a verifiable reward mechanism. The correctness reward measures grounding precision, while the calibration reward assesses uncertainty. For clarity, the reference model is omitted.
• Figure 3: Visualization of the confidence distribution output by HyperClick. We inject the coordinates on the interface into the assistant's generation and enforce it to continue to output the confidence for the click position. The darker the color, the higher the confidence value.