Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos

TL;DR

The paper tackles the challenge of scaling Vision-Language-Action pretraining for dexterous robotic manipulation by converting unannotated real-life egocentric hand videos into structured VLA data with 3D hand and camera motion labels. It introduces a fully automatic pipeline (3D motion labeling, atomic action segmentation, and instruction labeling) to generate a large hand-centric VLA dataset (1M episodes, 26M frames) and pairs this with a VLA model that combines a VLM backbone and a diffusion-based action predictor, trained end-to-end and fine-tuned on real robot data. Empirical results show strong zero-shot generalization to unseen real environments and notable gains when fine-tuned on modest robot data; data diversity and trajectory-guided episode construction are critical for performance gains. The work demonstrates favorable data-scaling behavior, with higher pretraining data volume yielding improved task success and generalization, and it lays a foundation for scalable embodied intelligence through web-scale human videos, with plans to open-source datasets and models.

Abstract

This paper presents a novel approach for pretraining robotic manipulation Vision-Language-Action (VLA) models using a large corpus of unscripted real-life video recordings of human hand activities. Treating human hand as dexterous robot end-effector, we show that "in-the-wild" egocentric human videos without any annotations can be transformed into data formats fully aligned with existing robotic V-L-A training data in terms of task granularity and labels. This is achieved by the development of a fully-automated holistic human activity analysis approach for arbitrary human hand videos. This approach can generate atomic-level hand activity segments and their language descriptions, each accompanied with framewise 3D hand motion and camera motion. We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames. This training data covers a wide range of objects and concepts, dexterous manipulation tasks, and environment variations in real life, vastly exceeding the coverage of existing robot data. We design a dexterous hand VLA model architecture and pretrain the model on this dataset. The model exhibits strong zero-shot capabilities on completely unseen real-world observations. Additionally, fine-tuning it on a small amount of real robot action data significantly improves task success rates and generalization to novel objects in real robotic experiments. We also demonstrate the appealing scaling behavior of the model's task performance with respect to pretraining data scale. We believe this work lays a solid foundation for scalable VLA pretraining, advancing robots toward truly generalizable embodied intelligence.

Figures (20)

• Figure 1: We present a pretraining approach for robotic Vision-Language-Action (VLA) models by transforming unstructured real-life videos of human activity into structured V-L-A formats aligned with existing robot data. The pretrained model demonstrates strong zero-shot hand action prediction in unseen environments and can be effectively fine-tuned with dexterous robot hand data for real-world tasks, showing robust generalization to new objects and environments.
• Figure 2: Overview of our holistic human activity analysis framework, which transforms unscripted real-life human videos into V-L-A episodes of human hands aligned with typical robotic data via three stages: (a) 3D motion labeling, reconstructing metric-scale 3D hand and camera trajectories; (b) atomic action segmentation, dividing videos into atomic-level clips; and (c) instruction labeling, employing GPT to annotate action instructions for each clip.
• Figure 2: Ablation study on hand action prediction using different episode construction strategies. $\dag$: Pretrained using a subset of 350K episodes for efficiency.
• Figure 3: Our VLA model architecture. It consists of a VLM backbone and a diffusion action expert. The VLM receives visual and linguistic instructions, as well as the camera FoV, and outputs a cognition feature that guides the action expert for future action chunk prediction. The action expert additionally receives the current state of the end effector and valid action masks for iterative action denoising via causal attention.
• Figure 4: Visual diversity across VLA datasets. (a) Image feature similarity with OpenImages kuznetsova2020open as the number of episodes varies. $\star$ marks the full dataset's similarity. (b) t-SNE visualization of image features.