OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

TL;DR

OmegaUse introduces a general-purpose GUI agent with a parameter-efficient MoE backbone for autonomous task execution across mobile and desktop environments. It pairs a high-quality grounding data pipeline with a decoupled two-stage training strategy (SFT then GRPO) and a unified cross-terminal action space to achieve strong cross-platform performance. The paper reports SOTA results on ScreenSpot-V2 (96.3%) and leading AndroidControl performance (79.1% SR), plus competitive OS-Nav results (ChiM-Nav 74.24% SR, Ubu-Nav 55.9% average), and contributes OS-Nav as an offline benchmark for cross-environment GUI navigation. Together, these components demonstrate OmegaUse’s effectiveness in grounding and planning across diverse digital ecosystems, enabling robust, efficient autonomous GUI interaction with practical implications for real-world agent deployment.

Abstract

Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.

Figures (3)

• Figure 1: An overview of OmegaUse's core capabilities in realistic GUI scenarios.
• Figure 2: The overall architecture of the OmegaUse framework. The pipeline proceeds through four distinct layers: (1) a hybrid data processing stage integrating automated LLM-assisted annotation and human-in-the-loop refinement; (2) SFT of an MoE foundation model; (3) decoupled RL using GRPO with tailored rewards for grounding and navigation tasks; and (4) final deployment of the optimized agents across diverse application environments.
• Figure 3: Overview of the Exploration-driven (Bottom-up) data construction pipeline. (a) Triples Collection: Gathering raw interaction primitives $< pre\_state, action, post\_state >$ through autonomous application exploration. (b) State Transition Graph Construction: Organizing interaction traces into a structured graph with MLLM-based semantic clustering to merge redundant UI states. (c) Trajectory Extraction: Sampling diverse execution paths while enriching them with natural language task goals and step-wise action interpretations.