Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration

Qingni Wang, Yue Fan, Xin Eric Wang

TL;DR

SafeGround addresses the risk of costly, irreversible errors in GUI grounding by introducing a distribution-aware, uncertainty-calibrated framework. It builds a spatial uncertainty representation from multiple stochastic grounding samples and uses Learn-Then-Test calibration to select a test-time threshold that controls the false discovery rate with finite-sample guarantees ($FDR$). The approach supports selective prediction and cascading inference, allowing safe execution for low-uncertainty cases and escalation to stronger models for high-uncertainty cases. Empirical results on ScreenSpot-Pro show improved discrimination of correct vs incorrect groundings and notable system-level accuracy gains across several GUI grounding models.

Abstract

Graphical User Interface (GUI) grounding aims to translate natural language instructions into executable screen coordinates, enabling automated GUI interaction. Nevertheless, incorrect grounding can result in costly, hard-to-reverse actions (e.g., erroneous payment approvals), raising concerns about model reliability. In this paper, we introduce SafeGround, an uncertainty-aware framework for GUI grounding models that enables risk-aware predictions through calibrations before testing. SafeGround leverages a distribution-aware uncertainty quantification method to capture the spatial dispersion of stochastic samples from outputs of any given model. Then, through the calibration process, SafeGround derives a test-time decision threshold with statistically guaranteed false discovery rate (FDR) control. We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark. Experimental results show that our uncertainty measure consistently outperforms existing baselines in distinguishing correct from incorrect predictions, while the calibrated threshold reliably enables rigorous risk control and potentials of substantial system-level accuracy improvements. Across multiple GUI grounding models, SafeGround improves system-level accuracy by up to 5.38\% percentage points over Gemini-only inference.

SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration

TL;DR

SafeGround addresses the risk of costly, irreversible errors in GUI grounding by introducing a distribution-aware, uncertainty-calibrated framework. It builds a spatial uncertainty representation from multiple stochastic grounding samples and uses Learn-Then-Test calibration to select a test-time threshold that controls the false discovery rate with finite-sample guarantees (). The approach supports selective prediction and cascading inference, allowing safe execution for low-uncertainty cases and escalation to stronger models for high-uncertainty cases. Empirical results on ScreenSpot-Pro show improved discrimination of correct vs incorrect groundings and notable system-level accuracy gains across several GUI grounding models.

Abstract

Graphical User Interface (GUI) grounding aims to translate natural language instructions into executable screen coordinates, enabling automated GUI interaction. Nevertheless, incorrect grounding can result in costly, hard-to-reverse actions (e.g., erroneous payment approvals), raising concerns about model reliability. In this paper, we introduce SafeGround, an uncertainty-aware framework for GUI grounding models that enables risk-aware predictions through calibrations before testing. SafeGround leverages a distribution-aware uncertainty quantification method to capture the spatial dispersion of stochastic samples from outputs of any given model. Then, through the calibration process, SafeGround derives a test-time decision threshold with statistically guaranteed false discovery rate (FDR) control. We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark. Experimental results show that our uncertainty measure consistently outperforms existing baselines in distinguishing correct from incorrect predictions, while the calibrated threshold reliably enables rigorous risk control and potentials of substantial system-level accuracy improvements. Across multiple GUI grounding models, SafeGround improves system-level accuracy by up to 5.38\% percentage points over Gemini-only inference.
Paper Structure (49 sections, 1 theorem, 24 equations, 20 figures, 6 tables, 1 algorithm)

This paper contains 49 sections, 1 theorem, 24 equations, 20 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Grounding
  4. Uncertainty Estimation
  5. Learn then Test Calibration
  6. Methodology
  7. Problem Formulation and Notations
  8. Method Overview
  9. Uncertainty Quantification
  10. Sampling-Based Spatial Distribution Construction
  11. Uncertainty Measurement 1. Top-Candidate Ambiguity (TA).
  12. Uncertainty Measurement 2. Informational Dispersion (IE).
  13. Uncertainty Measurement 3. Concentration Deficit (CD).
  14. Combined Uncertainty Score.
  15. Uncertainty Calibration for Selective Prediction
  16. ...and 34 more sections

Key Result

Lemma 3.1

Let $X \sim \mathrm{Bin}(n,p)$ be the number of successes in $n$ i.i.d. Bernoulli trials with success probability $p$. For any $\delta\in(0,1)$, define the Clopper-Pearson confidence interval where $\mathrm{Beta}^{-1}(q;a,b)$ denotes the $q$-quantile from a beta distribution with shape parameters $a$ and $b$. Then the interval has (at least) nominal coverage:

Figures (20)

  • Figure 1: While existing models may commit costly errors on hard-to-undo actions (e.g., checkout), SafeGround detects high uncertainty and defers the decision via cascading. This mechanism explicitly limits the risk of erroneous actions to a user-specified tolerance.
  • Figure 2: Overview of SafeGround. Given a GUI input, the model performs multiple stochastic grounding samples to estimate predictive uncertainty. An uncertainty threshold $\tau$ is calibrated on a held-out set under a user-specified risk level (i.e, the maximum error rate). At test time, predictions with uncertainty $\le \tau$ are executed directly, while high-uncertainty cases are abstained or cascaded. Low-uncertainty cases exhibit concentrated region scores, low entropy, and low variance, whereas high-uncertainty cases show dispersed predictions and trigger safety-aware deferral.
  • Figure 3: Test-time FDR (mean$\pm$std) on the ScreenSpot-Pro dataset under different risk levels.
  • Figure 4: Test-time power (mean) of our $U_{\text{COM}}$ and PC baseline on the ScreenSpot-Pro dataset under different risk levels.
  • Figure 5: Cascading rate (fraction of test samples deferred to Gemini) across different risk levels.
  • ...and 15 more figures

Theorems & Definitions (1)

  • Lemma 3.1: Clopper--Pearson interval clopper1934use