HRT1: One-Shot Human-to-Robot Trajectory Transfer for Mobile Manipulation

TL;DR

HRT1 addresses the challenge of teaching a mobile manipulator to imitate a human demonstration from a single RGB-D video without task-specific training. It combines hand-pose-based grasp transfer in UGCS, object-centric trajectory alignment with BundleSDF, and a two-stage trajectory optimization (base_pose and joint_trajectory) to robustly execute tasks in new environments. The approach yields high grasp success and task completion across 16 tasks, outperforming a baseline that relies on object-pose trajectories, and demonstrates robust performance under occlusions and pose variations. This work advances one-shot, training-free imitation for mobile manipulation with practical implications for rapid robot deployment in unstructured settings.

Abstract

We introduce a novel system for human-to-robot trajectory transfer that enables robots to manipulate objects by learning from human demonstration videos. The system consists of four modules. The first module is a data collection module that is designed to collect human demonstration videos from the point of view of a robot using an AR headset. The second module is a video understanding module that detects objects and extracts 3D human-hand trajectories from demonstration videos. The third module transfers a human-hand trajectory into a reference trajectory of a robot end-effector in 3D space. The last module utilizes a trajectory optimization algorithm to solve a trajectory in the robot configuration space that can follow the end-effector trajectory transferred from the human demonstration. Consequently, these modules enable a robot to watch a human demonstration video once and then repeat the same mobile manipulation task in different environments, even when objects are placed differently from the demonstrations. Experiments of different manipulation tasks are conducted on a mobile manipulator to verify the effectiveness of our system

Figures (10)

• Figure 1: An example of a real-world scene where a human wearing a HoloLens2 device gives voice commands to capture RGB-D data for moving a chair. A Fetch mobile manipulator is used for the data capture process.
• Figure 2: The overview of the Video Information Extraction (VIE) module for extracting object and hand information from a human demonstration video.
• Figure 3: Illustration of the UGCS coordinates khargonkar2024robotfingerprint for the Shadow hand and the Allegro hand.
• Figure 4: Illustration of the human hand to the Fetch gripper grasp transfer pipeline: (a) Inferring the 6D pose and joint values for an articulated hand model from HaMeR pavlakos2024HaMeR. (b) Grasp transfer: establishing correspondence between the human hand and the Fetch gripper points via the unified gripper coordinate space khargonkar2024robotfingerprint.
• Figure 5: An overview of the trajectory alignment pipeline within an object-centric reference frame using BundleSDF Wen2023-bundleSDF. This module is specifically designed to preserve consistency in demonstrations across diverse scene configurations and varying object poses.