Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

Hongzhe Bi, Lingxuan Wu, Tianwei Lin, Hengkai Tan, Zhizhong Su, Hang Su, Jun Zhu

TL;DR

H-RDT presents a two-stage, human-pretraining approach that leverages large-scale egocentric 3D hand-pose data to bootstrap robotic bimanual manipulation. By adopting a diffusion-transformer backbone with flow matching and modular action adapters, the model transfers manipulation priors across diverse robot morphologies. Across extensive simulation and real-world experiments, H-RDT consistently outperforms state-of-the-art baselines in single-task, multi-task, and few-shot settings, demonstrating strong cross-embodiment generalization. The work highlights human manipulation priors as a scalable foundation for efficient, cross-platform robotic learning with substantial practical impact.

Abstract

Imitation learning for robotic manipulation faces a fundamental challenge: the scarcity of large-scale, high-quality robot demonstration data. Recent robotic foundation models often pre-train on cross-embodiment robot datasets to increase data scale, while they face significant limitations as the diverse morphologies and action spaces across different robot embodiments make unified training challenging. In this paper, we present H-RDT (Human to Robotics Diffusion Transformer), a novel approach that leverages human manipulation data to enhance robot manipulation capabilities. Our key insight is that large-scale egocentric human manipulation videos with paired 3D hand pose annotations provide rich behavioral priors that capture natural manipulation strategies and can benefit robotic policy learning. We introduce a two-stage training paradigm: (1) pre-training on large-scale egocentric human manipulation data, and (2) cross-embodiment fine-tuning on robot-specific data with modular action encoders and decoders. Built on a diffusion transformer architecture with 2B parameters, H-RDT uses flow matching to model complex action distributions. Extensive evaluations encompassing both simulation and real-world experiments, single-task and multitask scenarios, as well as few-shot learning and robustness assessments, demonstrate that H-RDT outperforms training from scratch and existing state-of-the-art methods, including Pi0 and RDT, achieving significant improvements of 13.9% and 40.5% over training from scratch in simulation and real-world experiments, respectively. The results validate our core hypothesis that human manipulation data can serve as a powerful foundation for learning bimanual robotic manipulation policies.

H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation

TL;DR

H-RDT presents a two-stage, human-pretraining approach that leverages large-scale egocentric 3D hand-pose data to bootstrap robotic bimanual manipulation. By adopting a diffusion-transformer backbone with flow matching and modular action adapters, the model transfers manipulation priors across diverse robot morphologies. Across extensive simulation and real-world experiments, H-RDT consistently outperforms state-of-the-art baselines in single-task, multi-task, and few-shot settings, demonstrating strong cross-embodiment generalization. The work highlights human manipulation priors as a scalable foundation for efficient, cross-platform robotic learning with substantial practical impact.

Abstract

Imitation learning for robotic manipulation faces a fundamental challenge: the scarcity of large-scale, high-quality robot demonstration data. Recent robotic foundation models often pre-train on cross-embodiment robot datasets to increase data scale, while they face significant limitations as the diverse morphologies and action spaces across different robot embodiments make unified training challenging. In this paper, we present H-RDT (Human to Robotics Diffusion Transformer), a novel approach that leverages human manipulation data to enhance robot manipulation capabilities. Our key insight is that large-scale egocentric human manipulation videos with paired 3D hand pose annotations provide rich behavioral priors that capture natural manipulation strategies and can benefit robotic policy learning. We introduce a two-stage training paradigm: (1) pre-training on large-scale egocentric human manipulation data, and (2) cross-embodiment fine-tuning on robot-specific data with modular action encoders and decoders. Built on a diffusion transformer architecture with 2B parameters, H-RDT uses flow matching to model complex action distributions. Extensive evaluations encompassing both simulation and real-world experiments, single-task and multitask scenarios, as well as few-shot learning and robustness assessments, demonstrate that H-RDT outperforms training from scratch and existing state-of-the-art methods, including Pi0 and RDT, achieving significant improvements of 13.9% and 40.5% over training from scratch in simulation and real-world experiments, respectively. The results validate our core hypothesis that human manipulation data can serve as a powerful foundation for learning bimanual robotic manipulation policies.

Paper Structure

This paper contains 45 sections, 4 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning-based Robotic Manipulation
  4. Learning from Egocentric Human Manipulation
  5. Method
  6. Problem Formulation
  7. Overview
  8. Human Action Representation Design
  9. Two-Stage Training Paradigm
  10. Stage 1: Human Data Pre-training
  11. Stage 2: Cross-Embodiment Fine-tuning
  12. H-RDT Architecture
  13. Flow Matching for Action Generation
  14. Network Architecture
  15. Experiments
  16. ...and 30 more sections

Figures (3)

  • Figure 1: H-RDT framework. Our approach consists of two main stages: (1) pre-training on large-scale human manipulation data with 48-dimensional hand pose representations, and (2) cross-embodiment fine-tuning with modular action encoders and decoders adapted to specific robot action spaces.
  • Figure 2: Cross-embodiment multi-task performance on RoboTwin 2.0 tasks.
  • Figure 3: Task definition of real-world experiments.