GEM: Gaussian Embedding Modeling for Out-of-Distribution Detection in GUI Agents

TL;DR

This work addresses the challenge of out-of-distribution detection for GUI agents powered by multimodal LLMs, where traditional detectors struggle due to complex, evolving embedding spaces. It introduces GEM, a Gaussian Mixture Model-based method that operates on distances of input embeddings to a learned centroid, with component-wise boundaries determined by BIC and standard deviation multipliers. Across eight GUI datasets spanning smartphone, computer, and web platforms, GEM yields an average improvement of 23.7 percentage points in accuracy over the best baselines, with modest training and inference overhead. The approach also demonstrates potential end-to-end gains by leveraging cloud assistance for OOD samples and generalizes across multiple backbones, highlighting its practical value for safer and more reliable GUI agents.

Abstract

Graphical user interface (GUI) agents have recently emerged as an intriguing paradigm for human-computer interaction, capable of automatically executing user instructions to operate intelligent terminal devices. However, when encountering out-of-distribution (OOD) instructions that violate environmental constraints or exceed the current capabilities of agents, GUI agents may suffer task breakdowns or even pose security threats. Therefore, effective OOD detection for GUI agents is essential. Traditional OOD detection methods perform suboptimally in this domain due to the complex embedding space and evolving GUI environments. In this work, we observe that the in-distribution input semantic space of GUI agents exhibits a clustering pattern with respect to the distance from the centroid. Based on the finding, we propose GEM, a novel method based on fitting a Gaussian mixture model over input embedding distances extracted from the GUI agent that reflect its capability boundary. Evaluated on eight datasets spanning smartphones, computers, and web browsers, our method achieves an average accuracy improvement of 23.70\% over the best-performing baseline while only increasing training time by 4.9\% and testing time by 6.5\%. We also experimentally demonstrate that GEM can improve the step-wise success rate by 9.40\% by requesting assistance from the cloud model when encountering OOD samples. Analysis verifies the generalization ability of our method through experiments on nine different backbones. The codes are available at https://github.com/Wuzheng02/GEM-OODforGUIagents.

Figures (10)

• Figure 1: Comparison between an agent w/ and w/o OOD detection when facing an OOD instruction. Also illustrated are the OOD scenarios and the challenges of OOD detection for GUI agents.
• Figure 2: The output embeddings of the OS-Kairos and AITZ are visualized. Most of the samples are confused.
• Figure 3: The multimodal embeddings of the AITZ dataset show a clustered distribution pattern around the centroid.
• Figure 4: AUROC for OOD detection using embeddings from different layers of the GUI agent.
• Figure 5: GUI agent performance across different cluster distances.