ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands

TL;DR

This work tackles dexterous GUI manipulation by addressing the limitations of discrete-action GUI agents in performing continuous, on-the-fly drags. It introduces ShowUI-$\pi$, a lightweight flow-based model that unifies discrete clicks and continuous drags into a single action space, using a flow-matching action expert atop a vision-language backbone to generate smooth trajectories. The ScreenDrag benchmark provides 505 real-world drag tasks across five domains and dual online/offline evaluation protocols, with a training corpus of 20K dense trajectories. Empirically, ShowUI-$\pi$ achieves state-of-the-art online performance (26.98% overall) with 450M parameters and demonstrates robust trajectory modeling, highlighting the feasibility and value of continuous-control GUI agents. The work lays a foundation for human-like dexterity in digital environments and points to future scaling and planning enhancements to further close the gap with human performance.

Abstract

Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-$π$, the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-$π$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi.

Figures (18)

• Figure 1: Drag refers to a continuous interaction where the cursor maintains contact with the UI element while moving along a trajectory, rather than a single discrete click. Left: Visualization of ScreenDrag data domains.Right: ShowUI-$\pi$ is a lightweight flow-based generative model for GUI Automation that handles dragging actions requiring on-the-fly observation, such as drawing and Captcha solving. Given a query, ShowUI-$\pi$ efficiently generates corresponding continuous trajectory from streaming visual observations.
• Figure 2: ScreenDrag Automated Data Collection Pipeline. ScreenDrag automated data generation pipeline for continuous trajectory-based GUI interaction data. The pipeline includes three stages: (i) Element Parsing: The software application UI is parsed with UI Automation of Windows SDK in order to retrieve the UI element metadata. (ii) Task Proposal: Given the UI element metadata, an LLM will be prompted to generate a drag instruction, the expected metadata change and the drag code with dense trajectory. (iii) Trajectory Synthesis: The drag code will be executed in the software environment. A rule-based verifier will check the parsed metadata from UI states before and after the drag to ensure that the metadata change satisfies the expectation.
• Figure 3: ScreenDrag Data Distribution. The inner ring indicates the five equally distributed domains. The outer ring demonstrates per-category breakdowns with shares of the full dataset.
• Figure 4: Comparison between offline evaluation and online evaluation pipelines of ScreenDrag:(i) Offline evaluation is based on the distance of prediction and ground-truth in independent trunks; (ii) Online evaluation is incremental over sequential trunks, and based on the final outcome.
• Figure 5: Overview of ShowUI-$\pi$. Given a task query and visual observations, the model first processes them through the VLM to obtain intermediate hidden states, which are then attended by the action expert. During interaction, the predicted actions update the environment, the next observation is encoded, and a new action chunk is produced—enabling fine-grained, closed-loop cursor control.