GhostUI: Unveiling Hidden Interactions in Mobile UI

TL;DR

GhostUI addresses the gap in mobile UI understanding by introducing a dedicated hidden-interaction dataset collected from 81 Android apps, covering six gesture types and paired before/after states. The authors design a three-phase data-collection pipeline (automated probing, validation, and task contextualization) and provide a simplified view hierarchy to improve multimodal reasoning. Experiments with GPT-4o and Qwen2.5-VL show that fine-tuning on GhostUI increases gesture prediction accuracy and improves UI transition descriptions, highlighting the importance of expanded action spaces and multimodal context. The work demonstrates practical potential for guiding users and informing interaction design, while outlining limitations and future directions such as platform expansion and task-level evaluation. Overall, GhostUI lays a foundation for training mobile agents to recognize and act on hidden gestures, enabling more capable and user-friendly automation in real-world apps.

Abstract

Modern mobile applications rely on hidden interactions--gestures without visual cues like long presses and swipes--to provide functionality without cluttering interfaces. While experienced users may discover these interactions through prior use or onboarding tutorials, their implicit nature makes them difficult for most users to uncover. Similarly, mobile agents--systems designed to automate tasks on mobile user interfaces, powered by vision language models (VLMs)--struggle to detect veiled interactions or determine actions for completing tasks. To address this challenge, we present GhostUI, a new dataset designed to enable the detection of hidden interactions in mobile applications. GhostUI provides before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions, allowing VLMs to better recognize concealed gestures and anticipate post-interaction states. Quantitative evaluations with VLMs show that models fine-tuned on GhostUI outperform baseline VLMs, particularly in predicting hidden interactions and inferring post-interaction screens, underscoring GhostUI's potential as a foundation for advancing mobile task automation.

Figures (13)

• Figure 1: Overview of the UI Probing Tool Operation. The system identifies interactive UI elements through view hierarchy parsing, tracks elements across dynamic screens using path-based identification, executes diverse touch gestures, and monitors UI states to capture ephemeral changes.
• Figure 2: Overview of Validation Tool Interface for manual annotation and filtering of collected interaction data. (A) Screenshot comparison panel displaying temporal UI states (before, before_annotated with red outline indicating the target element) to facilitate visual change detection and outcome verification. (B) Decision panel for assessing interaction validity and determining hidden nature. (C) Visual element labeling panel for categorization of UI components within the target element's bounding box. In this example, only media is selected as it represents the visual content within the target area.
• Figure 3: Examples of Hidden Interaction across six gesture types: tap, double tap, long press, swipe, scroll, and pinch. For each gesture type, paired before and after screenshots from real mobile apps illustrate the visual effect of interacting with a specific UI component (highlighted in orange in the before state). These examples illustrate the visual transitions that occur as a result of specific gestures applied to targeted elements.
• Figure 4: Structure of the GhostUI, showing the comprehensive documentation of hidden interaction. Before-and-After screenshots with view hierarchies (during state only included for long press and pinch gestures), action, app metadata, and a task description.
• Figure 5: Element Label Co-occurrences (left) and Gesture-Element Distributions by element type (right). The heatmap on the left shows how often different visual labels (e.g., border, whitespace, icon, media, text) overlap within the same bounding box, revealing frequent single-label regions (diagonal entries) and multi-label patterns (off-diagonal cells). The heatmap on the right displays each gesture type’s relative association with hidden vs. open interactions across element labels, highlighting specific label–gesture pairings more likely to indicate hidden interactions.