UI-Venus Technical Report: Building High-performance UI Agents with RFT

TL;DR

UI-Venus proposes a native UI agent that operates from screenshots using a multimodal LLM and attains state-of-the-art results in UI grounding and navigation through reinforcement finetune with GRPO. It introduces a data-cleaning-centric training pipeline, a Self-Evolving Trajectory History Alignment mechanism, and Sparse Action Enhancement to improve planning and rare-action learning, achieving SOTA on multiple benchmarks with open-source releases. The approach leverages ~107k high-quality grounding samples and ~350k navigation traces, trained on Qwen2.5-VL, demonstrating strong generalization across mobile, desktop, and web interfaces and real-time online navigation. These contributions advance end-to-end UI agents, offering reproducible, scalable methods for robust UI interaction tasks and encouraging further research in community-driven data and methods.

Abstract

We present UI-Venus, a native UI agent that takes only screenshots as input based on a multimodal large language model. UI-Venus achieves SOTA performance on both UI grounding and navigation tasks using only several hundred thousand high-quality training samples through reinforcement finetune (RFT) based on Qwen2.5-VL. Specifically, the 7B and 72B variants of UI-Venus obtain 94.1% / 50.8% and 95.3% / 61.9% on the standard grounding benchmarks, i.e., Screenspot-V2 / Pro, surpassing the previous SOTA baselines including open-source GTA1 and closed-source UI-TARS-1.5. To show UI-Venus's summary and planing ability, we also evaluate it on the AndroidWorld, an online UI navigation arena, on which our 7B and 72B variants achieve 49.1% and 65.9% success rate, also beating existing models. To achieve this, we introduce carefully designed reward functions for both UI grounding and navigation tasks and corresponding efficient data cleaning strategies. To further boost navigation performance, we propose Self-Evolving Trajectory History Alignment & Sparse Action Enhancement that refine historical reasoning traces and balances the distribution of sparse but critical actions, leading to more coherent planning and better generalization in complex UI tasks. Our contributions include the publish of SOTA open-source UI agents, comprehensive data cleaning protocols and a novel self-evolving framework for improving navigation performance, which encourage further research and development in the community. Code is available at https://github.com/inclusionAI/UI-Venus.

Figures (13)

• Figure 1: UI-Venus achieves SOTA performance across multiple UI grounding and navigation benchmarks.
• Figure 2: Executions on typical grounding and navigation tasks of UI-Venus. a) The instruction and the screenshot are needed for UI-Venus-Ground to output the corresponding coordinates; b) For navigation tasks, historical context (thought-action pairs) are essential for UI-Venus-Navi, which will generate the thinking content and model action. The historical context will be updated after each step finished.
• Figure 3: The overview of the proposed Self-Evolving Trajectory History Alignment & Enhancement process, applied between training epochs. The process consists of two key components: 1) Trajectory History Alignment refines the historical context for each trajectory step. The model executes multiple rollouts to generate candidate thought–action pairs, applying an Action Exact Match filter to retain only those whose predicted actions match ground-truth actions. The corresponding thoughts are collected in pool $\mathcal{C}$ and subsequently replace the original thoughts in the historical context, creating an optimized trajectory history for the next training epoch. 2) Sparse Action Enhancement focuses on samples with sparse actions. Multiple variants are constructed by combining different rollout generated thoughts that lead to the same sparse action, effectively increasing the representation of these sparse but critical operations in the training distribution.
• Figure 4: Action type distribution in our mobile training data, showing a long-tailed profile with several low-frequency (sparse) actions.
• Figure 5: Grounding example for desktop professional software Blender. The instruction is "Increase Z axis" and the bounding box result is displayed as a dotted red box