ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flow

TL;DR

ViSA-Flow tackles the data bottleneck in robot manipulation by learning a semantic action flow representation from unlabeled human videos and transferring this knowledge to robots with limited demonstrations. It combines semantic grounding, temporal hand-object tracking, and flow-conditioned encoding into a ViSA-Flow intermediate representation, learned via a two-stage transformer-based pipeline: pretraining on human videos to learn a dynamics prior, then fine-tuning on robot data for policy adaptation. Experimental results on CALVIN and real-world tasks show state-of-the-art performance in low-data regimes and strong transfer from human observations to robotic execution. Ablations confirm the critical roles of pretraining, segmentation/tracking, and accurate semantic grounding in enabling robust long-horizon manipulation.

Abstract

One of the central challenges preventing robots from acquiring complex manipulation skills is the prohibitive cost of collecting large-scale robot demonstrations. In contrast, humans are able to learn efficiently by watching others interact with their environment. To bridge this gap, we introduce semantic action flow as a core intermediate representation capturing the essential spatio-temporal manipulator-object interactions, invariant to superficial visual differences. We present ViSA-Flow, a framework that learns this representation self-supervised from unlabeled large-scale video data. First, a generative model is pre-trained on semantic action flows automatically extracted from large-scale human-object interaction video data, learning a robust prior over manipulation structure. Second, this prior is efficiently adapted to a target robot by fine-tuning on a small set of robot demonstrations processed through the same semantic abstraction pipeline. We demonstrate through extensive experiments on the CALVIN benchmark and real-world tasks that ViSA-Flow achieves state-of-the-art performance, particularly in low-data regimes, outperforming prior methods by effectively transferring knowledge from human video observation to robotic execution. Videos are available at https://visaflow-web.github.io/ViSAFLOW.

Figures (6)

• Figure 1: Humans and robots often share underlying atomic actions for similar tasks. Our framework leverages large-scale, unlabeled human videos by extracting weakly supervised semantic action flow priors (ViSA-Flow). This knowledge is distilled into a human policy and efficiently transferred to learn a corresponding robot policy.
• Figure 2: ViSA-Flow Architecture and Policy Learning Framework. (a) During pretraining, hand-object interaction masks are extracted from large-scale video frames and amplified via tracking to generate semantic flow representations. (b) In the finetuning stage, a multi-modal Transformer architecture conditions on the goal image, a sequence of RGB observation frames enhanced with pre-trained ViSA-Flow, language instructions and robot state. The Transformer predicts future visual states, low-level robot actions, and task progress using dedicated decoders.
• Figure 3: Datasets used for pretraining, finetuning, and evaluation. We pretrain on Something-Something-V2 with text labels and placeholders to extract semantic action flow. We finetune on 34 tasks across CALVIN environments A–C and evaluate zero-shot on environment D mees2022calvin, where the robot completes 5 consecutive subtasks in one sequence.
• Figure 4: The real-world experiment setup. We evaluate ViSA-Flow on two single-stage manipulation tasks and a two-stage long-horizon manipulation task.
• Figure 5: Qualitative results on the real world long-horizon task. We visualize the decoded ViSA-Flow prediction at ${\hat{z}}_{t+1}$ against the actual ViSA-Flow ${z}_{t+1}$ extracted from the next observation for four execution phases. Two rows correspond to the two subtasks that make up the long-horizon evaluation: (Top)Subtask 1 – MoveContainer;(Bottom)Subtask 2 – PickEggplant. Qualitatively, the model’s one-step predictions closely follow the true motion of the manipulator and task-relevant objects, even as the scene evolves across distinct interaction stages.