Beyond Clicking:A Step Towards Generalist GUI Grounding via Text Dragging

TL;DR

This work expands GUI grounding beyond click actions by introducing GUI-Drag, a large-scale dataset for text dragging, and ScreenDrag, a benchmark with three interface context levels and three metrics designed to evaluate text-dragging capability. It demonstrates that models trained with continual learning on GUI-Drag can significantly improve text-dragging performance while preserving existing click-grounding abilities. The results reveal a bias toward clicking in current models and validate the proposed data synthesis and evaluation framework as a step toward generalist GUI grounding. All resources are open-sourced to encourage broader adoption and future research.

Abstract

Graphical user interface (GUI) grounding, the process of mapping human instructions to GUI actions, serves as a fundamental basis to autonomous GUI agents. While existing grounding models achieve promising performance to simulate the mouse click action on various click-based benchmarks, another essential mode of mouse interaction, namely dragging, remains largely underexplored. Yet, dragging the mouse to select and manipulate textual content represents a prevalent and important usage in practical GUI scenarios. To narrow this gap, we first introduce GUI-Drag, a diverse dataset of 161K text dragging examples synthesized through a scalable pipeline. To support systematic and robust evaluation, we further construct ScreenDrag, a benchmark with 5,333 examples spanning three levels of interface context, together with three dedicated metrics designed for assessing text dragging capability. Models trained on GUI-Drag with an efficient continual training strategy achieve substantial improvements on ScreenDrag, while preserving the original click-based performance on ScreenSpot, ScreenSpot-v2, and OSWorld-G. Our work encourages further research on broader GUI grounding beyond just clicking and paves way toward a truly generalist GUI grounding model. All benchmark, data, checkpoints, and code are open-sourced and available at https://osu-nlp-group.github.io/GUI-Drag.

Figures (10)

• Figure 1: Illustration of the ScreenDrag benchmark and the task of text dragging. The left part shows three levels of interface context within the benchmark (examples in Appendix \ref{['app:interface_context_examples']}) and the right part shows the process of grounding the text selection by dragging.
• Figure 2: A screenshot with SOM in gray. In the top-left black box, the target text span is the first sentence, “Like … tools.”. Given the ground truth and predicted bboxes (The predicted start and end coordinates fall within bbox 0 and bbox 20, which we omit here to avoid clutter.), the $\text{B-Dist}$ is 3 according to Equation \ref{['eq:bdist']}. In the bottom-right box, the target span is the last sentence, “For … Word.”. Here, both predictions (blue and green) yield zero $\text{B-Dist}$, but only the green coordinates correctly capture the target span. The blue prediction fails because its $d_{\text{pixel}}$ does not satisfy the small threshold, whereas the green one succeed given the text snapping mechanism.
• Figure 3: Distribution of actions across explicit and implicit instructions across 5 models.
• Figure 4: Success rate on click-based benchmarks (x-axis; average score reported) and ScreenDrag (y-axis) with different proportion of Jedi data.
• Figure 5: Resolution size analysis of the benchmark.