VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos

TL;DR

This paper tackles the data bottleneck for training computer-use agents by introducing VideoAgentTrek, a scalable pipeline that mines structured GUI interaction trajectories from unlabeled screen-recorded videos. It hinges on Video2Action, an inverse dynamics module that first localizes GUI events with precise timing and then parameterizes each action (e.g., click coordinates, typed text), complemented by inner-monologue generation to guide planning. The authors demonstrate that mining 1.52 million interaction steps from 39,000 YouTube tutorials, followed by a two-stage pretraining and finetuning strategy, yields substantial gains on both online (OSWorld-Verified) and offline (AgentNetBench) benchmarks, and enables test-time scaling with longer planning horizons. This approach shows that passive internet videos can provide high-quality supervision at scale, offering a cost-effective alternative to manual annotation for training robust computer-use agents, with the ScreenFilter and Video2Action tools released for the community.

Abstract

Training computer-use agents requires massive amounts of GUI interaction data, but manually annotating action trajectories at scale is prohibitively expensive. We present VideoAgentTrek, a scalable pipeline that automatically mines training data from publicly available screen-recorded videos at web scale, eliminating the need for manual annotation. Our approach addresses a key challenge: raw videos contain implicit demonstrations but lack explicit action labels. To solve this, we develop Video2Action, an inverse dynamics module (IDM) with two components: (1) a video grounding model that detects and localizes GUI actions with precise temporal boundaries and context, and (2) an action-content recognizer that extracts structured parameters like click coordinates and typed text with high fidelity. Applied to 39,000 YouTube tutorial videos, our pipeline generates 1.52 million interaction steps automatically. We leverage this data through continued pretraining followed by supervised fine-tuning. On OSWorld-Verified, our approach improves task success rates from 9.3% (SFT-only baseline) to 15.8%, a 70% relative improvement. On AgentNetBench, step accuracy increases from 64.1% to 69.3%. Our results demonstrate that passive internet videos can be transformed into high-quality supervision for computer-use agents, providing a scalable alternative to expensive manual annotation.

Figures (12)

• Figure 1: Overview of VideoAgentTrek. (1) Video Collection: crawl screen-recorded tutorials and filter GUI footage with ScreenFilter. (2) Video2Action: an inverse dynamics module that first performs dense action-event detection to localize clips and assign action types, then action parameterization (e.g., click coordinates, typed text) to yield structured $(\text{screenshot}, \text{action}, \text{parameters})$ trajectories. (3) Agent Training: use the mined trajectories for continued pretraining and supervised finetuning of computer-use agents.
• Figure 2: Video candidate auto-discovery. From seed keywords and tags, we search and evaluate videos, expand to related videos and high-quality channels ($\geq$80% pass), and iteratively collect GUI-containing videos for VAT.
• Figure 3: Domain distribution.
• Figure 4: Overview of Video2Action: Given a screen-capture video (with optional subtitles), the module (1) detects GUI action events and segments clips, (2) parameterizes each action (type and arguments), and (3) generates step-level thoughts, yielding training-ready sequences of {action clip, action, thought}
• Figure 5: Experimental Results on OSWorld-Verified osworld_verified and AgentNetBench wang2025opencuaopenfoundationscomputeruse. VideoAgentTrek demonstrates significant improvements over baseline models, with test-time scaling providing additional performance gains