RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

TL;DR

RDT2 tackles data scarcity and cross-embodiment generalization in vision-language-action robotics by combining a 7B pretrained VLM with a three-stage training pipeline (RVQ discretization, flow-matching action learning, and diffusion distillation) on a large-scale, embodiment-agnostic UMI dataset. The approach enables zero-shot generalization across unseen objects, scenes, instructions, and hardware, and achieves state-of-the-art performance on dexterous and dynamic tasks like table tennis after fine-tuning. The work demonstrates scalable data-driven robotics, showing how increasing model and data scale yields predictable gains and highlighting an effective path toward open-vocabulary, cross-embodiment robotic agents with real-time inference. It also discusses practical deployment considerations, including safety guardrails and privacy, given the in-home data collection foundation.

Abstract

Vision-Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic datasets--over 10,000 hours of demonstrations in diverse families--using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis. See https://rdt-robotics.github.io/rdt2/ for more information.

Figures (17)

• Figure 1: Illustration of our UMI solution. We re-designed the UMI hardware for better consistency and reliability in large-scale data collection. As long as the same model of camera and gripper are installed, the policy trained on the data collected by our UMIs can be zero-shot transferred to various robot arms.
• Figure 2: A three-stage pipeline for training RDT2. In Stage 1, we pre-train a 7B VLM backbone with discretized action data for vision-language reasoning capabilities. Then, in Stage 2, we train a small diffusion action expert to generate continuous actions efficiently. For highly dynamic tasks, we introduce a third stage that distills the diffusion policy into a one-step generator, thereby enabling extremely rapid inference speed.
• Figure 3: Results of zero-shot experiments of RDT2. The error bar represents the standard error.
• Figure 4: Convergence curve of statistical success rate in repeated trials (Pick Task, RDT2-FM).
• Figure 5: Scaling laws of RDT2. Left: Training loss as a function of consumed tokens (non-repeating) under various model parameter scales. "Total" parameters includes vision encoders. Right: Training loss as a function of total model parameters under different amounts of training data (measured by tokens).