OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent

TL;DR

OS-Symphony is introduced, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation, featuring a Reflection-Memory Agent that utilizes milestone-driven long-term memory to enable trajectory-level self-correction, effectively mitigating visual context loss in long-horizon tasks.

Abstract

While Vision-Language Models (VLMs) have significantly advanced Computer-Using Agents (CUAs), current frameworks struggle with robustness in long-horizon workflows and generalization in novel domains. These limitations stem from a lack of granular control over historical visual context curation and the absence of visual-aware tutorial retrieval. To bridge these gaps, we introduce OS-Symphony, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation: (1) a Reflection-Memory Agent that utilizes milestone-driven long-term memory to enable trajectory-level self-correction, effectively mitigating visual context loss in long-horizon tasks; (2) Versatile Tool Agents featuring a Multimodal Searcher that adopts a SeeAct paradigm to navigate a browser-based sandbox to synthesize live, visually aligned tutorials, thereby resolving fidelity issues in unseen scenarios. Experimental results demonstrate that OS-Symphony delivers substantial performance gains across varying model scales, establishing new state-of-the-art results on three online benchmarks, notably achieving 65.84% on OSWorld.

Figures (11)

• Figure 1: Current limitations in CUA framework.
• Figure 2: Pipeline overview.OS-Symphony comprises three primary components: (1) The Orchestrator, acting as the system's brain, responsible for task understanding and action prediction; (2) Tool Agents, consisting of Grounder, Coder, and Searcher, where the Searcher retrieves up-to-date tutorials in a human-like manner; and (3) The Reflection-Memory Agent, which compresses execution trajectories to maintain long-term memory and facilitate trajectory-level reflection.
• Figure 3: Pipeline of RMA. At each step, RMA summarizes the previous action using pre- and post-action screenshots and the Orchestrator’s output, while evaluating the current GUI operation’s correctness. It then generates a reflection from all summaries and milestone screenshots, and determines whether the latest step is a milestone.
• Figure 4: The Pass@K results on OSWorld. All experiments are carried out with GPT-5 and 100 steps limit.
• Figure 5: Discussion on the impact of varying the maximum number of images in agent’s trajectory. All experiments are carried out with Qwen3-VL-32B-Instruct and UI-TARS-1.5-7B.