Learning from Online Videos at Inference Time for Computer-Use Agents

TL;DR

This work tackles the gap in computer-use agents’ domain-specific procedural knowledge by enabling inference-time learning from online video tutorials. It introduces a three-part framework: (1) retrieval and filtering of relevant videos, (2) conversion of videos into compact, visually grounded demonstration trajectories via a vision-language model, and (3) a two-stage in-context trajectory selection that provides the most helpful local guidance for the agent’s next action. Across OSWorld and WebArena, the approach consistently surpasses baselines that rely on textual tutorials or transcripts, demonstrating the value of visual information and dynamic trajectory selection for desktop GUI tasks. The findings suggest that abundant online videos can be distilled into actionable guidance that meaningfully improves agent performance, with significant implications for scalable, real-world automation. The method highlights practical gains in task efficiency and indicates avenues for further scaling through larger video corpora and refined trajectory construction.

Abstract

Computer-use agents can operate computers and automate laborious tasks, but despite recent rapid progress, they still lag behind human users, especially when tasks require domain-specific procedural knowledge about particular applications, platforms, and multi-step workflows. Humans can bridge this gap by watching video tutorials: we search, skim, and selectively imitate short segments that match our current subgoal. In this paper, we study how to enable computer-use agents to learn from online videos at inference time effectively. We propose a framework that retrieves and filters tutorial videos, converts them into structured demonstration trajectories, and dynamically selects trajectories as in-context guidance during execution. Particularly, using a VLM, we infer UI actions, segment videos into short subsequences of actions, and assign each subsequence a textual objective. At inference time, a two-stage selection mechanism dynamically chooses a single trajectory to add in context at each step, focusing the agent on the most helpful local guidance for its next decision. Experiments on two widely used benchmarks show that our framework consistently outperforms strong base agents and variants that use only textual tutorials or transcripts. Analyses highlight the importance of trajectory segmentation and selection, action filtering, and visual information, suggesting that abundant online videos can be systematically distilled into actionable guidance that improves computer-use agents at inference time. Our code is available at https://github.com/UCSB-NLP-Chang/video_demo.

Figures (16)

• Figure 1: Overview of our inference pipeline. Step 1 outputs a set of videos relevant to the query task. Step 2 yields a set of demonstration trajectories, each with a textual objective and a sequence of actions and observations. Step 3 dynamically selects a single trajectory to add in context before issuing each action, focusing the agent on the most helpful local guidance for its decision.
• Figure 1: Success rate on OSWorld. "Has Videos" denotes the subset of 211 tasks for which we find at least one relevant video.
• Figure 2: Prompt used to generate search queries for video tutorials.
• Figure 3: Performance on OSWorld with different numbers of videos per task.
• Figure 4: Ablation study on OSWorld.