DIRIGENt: End-To-End Robotic Imitation of Human Demonstrations Based on a Diffusion Model

TL;DR

DIRIGENt addresses the challenge of teaching humanoid robots from human demonstrations by directly generating joint values from RGB frames using diffusion, eliminating the need for explicit human–robot mappings. It introduces a novel dataset of matched human and robot poses to support end-to-end learning and demonstrates that RGB-conditioned diffusion yields accurate imitation and can outperform a CNN-based baseline in several settings. The approach integrates a forward-kinematics module to connect joint configurations with end-effector Cartesian positions and employs a joint-plus-Cartesian loss to resolve redundancy, achieving real-time inference on standard GPUs. Overall, the work demonstrates the practical potential of end-to-end diffusion for imitation learning and shows generalization to unseen humans and tasks, guided by targeted ablations.

Abstract

There has been substantial progress in humanoid robots, with new skills continuously being taught, ranging from navigation to manipulation. While these abilities may seem impressive, the teaching methods often remain inefficient. To enhance the process of teaching robots, we propose leveraging a mechanism effectively used by humans: teaching by demonstrating. In this paper, we introduce DIRIGENt (DIrect Robotic Imitation GENeration model), a novel end-to-end diffusion approach that directly generates joint values from observing human demonstrations, enabling a robot to imitate these actions without any existing mapping between it and humans. We create a dataset in which humans imitate a robot and then use this collected data to train a diffusion model that enables a robot to imitate humans. The following three aspects are the core of our contribution. First is our novel dataset with natural pairs between human and robot poses, allowing our approach to imitate humans accurately despite the gap between their anatomies. Second, the diffusion input to our model alleviates the challenge of redundant joint configurations, limiting the search space. And finally, our end-to-end architecture from perception to action leads to an improved learning capability. Through our experimental analysis, we show that combining these three aspects allows DIRIGENt to outperform existing state-of-the-art approaches in the field of generating joint values from RGB images.

Figures (5)

• Figure 1: The robot NICOL is on the right side with a human demonstrator on the left side. The robot matches the human's arm poses.
• Figure 2: Comparison of datasets: on the left is the EMIL dataset and on the right is the Direct Imitation Dataset.
• Figure 3: Our architecture. On the left side the differences between training and inference are shown as well as all inputs to the model, on the right side the neural architecture and the outputs are shown.
• Figure 4: The end effector position in the label, the model's prediction and pose estimation over 2400 frames normalized to be between 0 and 1 along the x-axis and the y-axis.
• Figure 5: On the left is the human demonstration, and on the right is the robotic imitation based on the generated joint configuration.