Emergence of Human to Robot Transfer in Vision-Language-Action Models

TL;DR

The paper investigates whether embedding diverse, embodied human video data into Vision-Language-Action (VLA) pretraining induces emergent human-to-robot transfer. It introduces a simple co-training approach that treats human demonstrations as another embodiment with the same objectives used for robot data, evaluating on generalization benchmarks across unseen scenes, objects, and tasks. Results show that transfer emerges with sufficiently diverse pretraining and can nearly double performance in some generalization settings, with embeddings becoming embodiment-agnostic as diversity grows. This suggests scalable, cross-embodiment learning for robotic foundation models and highlights the potential of embodied human data to expand open-world robot capabilities.

Abstract

Vision-language-action (VLA) models can enable broad open world generalization, but require large and diverse datasets. It is appealing to consider whether some of this data can come from human videos, which cover diverse real-world situations and are easy to obtain. However, it is difficult to train VLAs with human videos alone, and establishing a mapping between humans and robots requires manual engineering and presents a major research challenge. Drawing inspiration from advances in large language models, where the ability to learn from diverse supervision emerges with scale, we ask whether a similar phenomenon holds for VLAs that incorporate human video data. We introduce a simple co-training recipe, and find that human-to-robot transfer emerges once the VLA is pre-trained on sufficient scenes, tasks, and embodiments. Our analysis suggests that this emergent capability arises because diverse pretraining produces embodiment-agnostic representations for human and robot data. We validate these findings through a series of experiments probing human to robot skill transfer and find that with sufficiently diverse robot pre-training our method can nearly double the performance on generalization settings seen only in human data.

Figures (13)

• Figure 1: Emergence of human to robot transfer: We observe that the transfer from human data to robot policies scales with the size and diversity of VLA pre-training data. The x-axis represents the diversity of the pre-training robot dataset, and the yellow and blue lines show the finetuning performance with and without human embodiment data. While both increase, the gain from leveraging human data only appears beyond a certain pre-training scale. We evaluate on a suite of four generalization scenarios shown only in the human data.
• Figure 2: Per-task improvement from human data: We plot the difference in performance between policies fine-tuned with robot + human data versus robot-only data, isolating the lift from human supervision. Gains are largest when pre-training spans diverse tasks, scenes, and embodiments, suggesting that broad pre-training improves transfer from human videos.
• Figure 3: Training mixture and benchmark. Our fine-tuning mix is evenly split between human data for generalization tasks and robot data for the nearest neighbor task. For each task we evaluate generalization to a new concept introduced only in human data 1) Scene generalization: We have robot data for tidying dressers and spice racks across many airbnbs and human data for an unseen apartment. 2) Object generalization: robot data covers bussing a table filled with trash and dinnerware and human data for a new set of objects. 3) Task generalization: robot data covers placing eggs into cartons, but human data introduces the new concept of sorting eggs by color.
• Figure 4: Model Architecture. We use the $\pi_{0.5}$ model. We finetune with a combination of high level sub-task prediction and low level action prediction on both human and robot data. The low level action prediction leverages relative end-effector actions aligned across human and robot.
• Figure 5: VLA representation of human and robot data. We plot the latent embeddings of our VLA by performing a TSNE analysis on mean-pooled tokens from the final layer of the VLM backbone. With no pre-training, it is clear that the model has disjoint representations between human and robot data. But as pretraining becomes more diverse, latent overlap increases, which correlates with performance on our generalization tasks.