OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks

TL;DR

OS-Marathon formalizes long-horizon, repetitive computer-use tasks within a POMDP framework and introduces the first benchmark spanning 2 domains to evaluate CUAs on extended workflows, totaling $N$ sub-workflows per task. The paper identifies critical failure modes of current SOTA CUAs—logical incoherence, grounding hallucinations, and loss of long-horizon consistency—and tackles them with a cost-efficient Few-shot Condensed Workflow Demonstration, which abstracts the workflow into a small, semantic key-step program guiding both global planning and per-sub-workflow execution. Empirical results show that conventional baselines struggle dramatically on these tasks (often 0% SR), while the condensed demonstration significantly improves Sub-Workflow Accuracy (SWA), particularly in web-based environments, though full horizon completion remains challenging due to context window and grounding constraints. Overall, the work lays a foundation for evaluating and improving CUAs on long-horizon repetitive tasks and points to future work on stronger constraint mechanisms and more scalable demonstrations to bridge the remaining gap to fully autonomous end-to-end performance.

Abstract

Long-horizon, repetitive workflows are common in professional settings, such as processing expense reports from receipts and entering student grades from exam papers. These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process. However, they are ideal for Computer-Use Agents (CUAs) due to their structured, recurring sub-workflows with logic that can be systematically learned. Identifying the absence of an evaluation benchmark as a primary bottleneck, we establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents. We then introduce a cost-effective method to construct a condensed demonstration using only few-shot examples to teach agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections. Extensive experiments demonstrate both the inherent challenges of these tasks and the effectiveness of our proposed method. Project website: https://os-marathon.github.io/.

Figures (15)

• Figure 1: Overview of . We investigate long-horizon, repetitive workflows (Top Left), where a Computer-Use Agent (CUA) iteratively processes a data collection via invariant sub-workflows. We introduce a benchmark spanning domains (Right) to evaluate state-of-the-art agents on these tasks. Furthermore, we propose a cost-effective few-shot condensed workflow demonstration strategy (Bottom Left) that teaches agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections.
• Figure 2: Examples of the execution environments in .
• Figure 3: Synthetic data pipeline. User profiles are synthesised via an LLM and subsequently rendered using a set of styled templates.
• Figure 4: A natural execution illustration for an example task in the expense report domain. The workflow involves iteratively populating a university expense web system using a student's profile and a diverse collection of receipts.
• Figure 5: Demonstration-guided CUA workflow on . The figure illustrates the complete process by which CUA executes long-horizon, repetitive desktop tasks under human guidance.