Learning Skills from Action-Free Videos

TL;DR

SOF introduces Skill Abstraction from Optical Flow to learn temporally extended, transferable skills directly from action-free videos. It builds a discrete latent skill space from optical-flow sequences, learns a Transformer-based skill policy conditioned on observations and instructions, and uses a Flow2Action module to convert flow plans into executable actions. Across multi-task, long-horizon, and cross-embodiment experiments, SOF outperforms pixel-space and single-step latent-action baselines, while remaining data-efficient and robust to variations in flow estimators. This work demonstrates that mid-level motion representations like optical flow can drive scalable robot learning from unlabeled visual data, enabling reusable skills and cross-domain transfer.

Abstract

Learning from videos offers a promising path toward generalist robots by providing rich visual and temporal priors beyond what real robot datasets contain. While existing video generative models produce impressive visual predictions, they are difficult to translate into low-level actions. Conversely, latent-action models better align videos with actions, but they typically operate at the single-step level and lack high-level planning capabilities. We bridge this gap by introducing Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from large collections of action-free videos. Our key idea is to learn a latent skill space through an intermediate representation based on optical flow that captures motion information aligned with both video dynamics and robot actions. By learning skills in this flow-based latent space, SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions. Experiments show that our approach consistently improves performance in both multitask and long-horizon settings, demonstrating the ability to acquire and compose skills directly from raw visual data.

Figures (11)

• Figure 1: Extracting skills from videos. Videos contain composable skills that appear across different tasks and scenes. Learning and planning in a skill space enables efficient multi-task learning and long-horizon planning. Learning skills from raw images (top) often overfits to visual appearance. Instead, we learn skills from optical flow (bottom), which captures motion patterns and better reflects the underlying actions.
• Figure 2: Skill Abstraction from Optical Flow (SOF). (a) Learn an action abstraction from optical flow using latent variable models to capture motion patterns across tasks. (b) Learn a skill predictor to perform policy learning in the skill space. (c) Given the first frame and an instruction, SOF generates a skill plan, decodes it into optical flow using a decoder, and infers actions using a lightweight Flow2Action module. The Flow2Action module can be either learned or calculated.
• Figure 3: Environmental setups: (a) MetaWorld is a simulation benchmark featuring a variety of manipulation tasks. We used it to evaluate multi-task performance and cross-embodiment generalization. (b) LIBERO is a simulation benchmark for lifelong robot learning. We use it to study multi-task and long-horizon performance. (c) BridgeData V2 is a real-world dataset of manipulation behaviors. We use it to evaluate our skills on diverse environments and tasks.
• Figure 4: Cross-embodiment transfer. Average success rates on MetaWorld: (a) Skill policy transfer. We train the skill abstraction with both Panda and Sawyer data. In the policy training stage, only one embodiment's data is used. Topline shows results trained on all tasks per embodiment. The results show that shared skill representation enable effective transfer across embodiments. (b) Cross-task transfer. We partition tasks into disjoint sets, A and B. The cross-task policy is trained on Sawyer data from A and Panda data from B, while the topline policy is trained on the full dataset. The results indicate successful transfer, even when a task is unseen for one embodiment.
• Figure 5: Skill Token Analysis: Each figure shows the optical flow plan that corresponds to the same skill token. (a) Different tasks and scenes involving similar motion patterns are grouped together (b) Visually distinct objects in the same scene, positioned differently, are grouped together due to shared motion (c) Visually diverse real-world scenes are grouped together by shared motion patterns.