A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulation

TL;DR

The paper tackles the data scarcity barrier in large behavior models for robot manipulation by conducting a comprehensive, large-scale empirical study of co-training across five heterogeneous data modalities. It deploys a vision-language-action framework with an Action Flow Transformer and analyzes three distinct co-training strategies, evaluating 89 policies on 4,000 hours of robot/human data and 50 million vision-language samples across 58,000 simulation rollouts and 2,835 real-world trials. Key findings show that diverse vision-language data and cross-embodiment robot data consistently improve generalization to distribution shifts, unseen tasks, and language following, while discrete action tokens provide little to no benefit; combining effective modalities yields cumulative performance gains and enables rapid adaptation via fine-tuning. The study also reveals that training exclusively on robot data can erode visiolinguistic understanding, whereas co-training preserves and enhances it, and that explicit chain-of-thought conditioning for action generation does not improve performance in their manipulation benchmarks. Overall, the work offers practical guidance for building scalable generalist robot policies and highlights the central role of rich vision-language pretraining in aligning perception with action.

Abstract

Large behavior models have shown strong dexterous manipulation capabilities by extending imitation learning to large-scale training on multi-task robot data, yet their generalization remains limited by the insufficient robot data coverage. To expand this coverage without costly additional data collection, recent work relies on co-training: jointly learning from target robot data and heterogeneous data modalities. However, how different co-training data modalities and strategies affect policy performance remains poorly understood. We present a large-scale empirical study examining five co-training data modalities: standard vision-language data, dense language annotations for robot trajectories, cross-embodiment robot data, human videos, and discrete robot action tokens across single- and multi-phase training strategies. Our study leverages 4,000 hours of robot and human manipulation data and 50M vision-language samples to train vision-language-action policies. We evaluate 89 policies over 58,000 simulation rollouts and 2,835 real-world rollouts. Our results show that co-training with forms of vision-language and cross-embodiment robot data substantially improves generalization to distribution shifts, unseen tasks, and language following, while discrete action token variants yield no significant benefits. Combining effective modalities produces cumulative gains and enables rapid adaptation to unseen long-horizon dexterous tasks via fine-tuning. Training exclusively on robot data degrades the visiolinguistic understanding of the vision-language model backbone, while co-training with effective modalities restores these capabilities. Explicitly conditioning action generation on chain-of-thought traces learned from co-training data does not improve performance in our simulation benchmark. Together, these results provide practical guidance for building scalable generalist robot policies.

Figures (33)

• Figure 1: Overview of the data, model architecture, and evaluation setup. Our policy is built on a pretrained vision-language model backbone combined with an Action Flow Transformer. It is trained on target robot data alongside heterogeneous co-training modalities, including standard vision-language data, dense language annotations for robot data, cross-embodiment robot data, human videos, and discrete robot action tokens. We evaluate policies in simulation on seen and unseen tasks, under nominal conditions and distribution shifts, and in the real-world for language following, and long-horizon dexterous manipulation.
• Figure 2: Overview of the training data. Our dataset comprises target robot data collected in both simulation and real-world, and five heterogeneous co-training data modalities: standard vision-language data for commonsense understanding, spatial reasoning, and object grounding; dense language annotations for robot data, generated via heuristic scripting and VLM-based captioning; cross-embodiment robot data capturing diverse robot morphologies and manipulation tasks; human videos, from which we derive either latent action tokens using a latent action model (LAM) or VLM-generated annotations; and discrete robot action tokens, including near-lossless FAST tokens and compact VQ-VAE tokens. These co-training modalities, together with the target robot data, constitute a unified dataset of $\sim$4,000 hours of manipulation data and 50M vision-language samples.
• Figure 3: Simulation and real-world evaluation. Policies are evaluated in simulation on 13 seen and 8 unseen tasks under nominal and distribution shift (DS) conditions, where DS introduces appearance changes (e.g., lighting, textures, distractors, camera parameters). Real-world evaluations include language-following experiments with seen objects, instruction generalization through paraphrasing, and unseen objects, as well as adaptation to unseen long-horizon dexterous tasks via fine-tuning.
• Figure 4: Simulation ablation of co-training data and strategies. Comparison of the no-co-training baseline with policies co-trained on a single data modality across sequential training phases. Policies are evaluated on seen and unseen tasks under nominal and distribution shift conditions (A--H denote data modalities).
• Figure 5: Real-world ablation of co-training data and strategies. Performance of the no-co-training baseline and policies co-trained with a single data modality across training phases, evaluated at language-following with seen objects, instruction generalization, and unseen objects (A--E denote data modalities).