ColorAgent: Building A Robust, Personalized, and Interactive OS Agent

TL;DR

ColorAgent tackles the challenge of turning OS agents into warm, proactive partners by enabling long-horizon environmental interaction and personalized user collaboration. It introduces a two-stage training regime (Step-wise RL and Self-Evolving Training) and a triad-based multi-agent framework (Knowledge Retrieval, Task Orchestration, Hierarchical Reflection) to achieve robustness and generalization. Empirical results on AndroidWorld and AndroidLab show state-of-the-art success rates of 77.2% and 50.7%, with additional gains in human-agent interaction benchmarks, illustrating practical impact. The work highlights the need for holistic evaluation and safe, collaborative OS agents, and lays a foundation for future research on evaluation paradigms, agent collaboration, and security.

Abstract

With the advancements in hardware, software, and large language model technologies, the interaction between humans and operating systems has evolved from the command-line interface to the rapidly emerging AI agent interactions. Building an operating system (OS) agent capable of executing user instructions and faithfully following user desires is becoming a reality. In this technical report, we present ColorAgent, an OS agent designed to engage in long-horizon, robust interactions with the environment while also enabling personalized and proactive user interaction. To enable long-horizon interactions with the environment, we enhance the model's capabilities through step-wise reinforcement learning and self-evolving training, while also developing a tailored multi-agent framework that ensures generality, consistency, and robustness. In terms of user interaction, we explore personalized user intent recognition and proactive engagement, positioning the OS agent not merely as an automation tool but as a warm, collaborative partner. We evaluate ColorAgent on the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2% and 50.7%, respectively, establishing a new state of the art. Nonetheless, we note that current benchmarks are insufficient for a comprehensive evaluation of OS agents and propose further exploring directions in future work, particularly in the areas of evaluation paradigms, agent collaboration, and security.

Figures (6)

• Figure 1: An overview of how the OS agent interacts with both the environment and the user.
• Figure 2: Our two-stage training framework for developing a powerful GUI model. The process begins with Step-wise Reinforcement Learning (left) to optimize the agent's core decision-making abilities on constructed training data. It then progresses to Self-evolving Training (right), a iterative loop that generates high-quality trajectory data to enable continuous improvement.
• Figure 3: Error type distribution of the single agent.
• Figure 4: The architecture of our multi-agent framework. It consists of a central Execution module supported by three key components: the Task Orchestration module decomposes complex user instructions into manageable atomic tasks and handles memory transfer; the Knowledge Retrieval module provides relevant external information; and the Hierarchical Reflection module delivers multi-level feedback for error correction.
• Figure 5: Training reward dynamics of different models.