Mitty: Diffusion-based Human-to-Robot Video Generation

TL;DR

Mitty tackles the challenge of learning robot manipulation directly from human demonstration videos by eliminating intermediate representations and leveraging a diffusion-transformer with in-context learning. It introduces a scalable paired-data synthesis pipeline to convert egocentric human videos into high-quality human–robot pairs and fuses human conditioning tokens with robot denoising tokens via bidirectional attention on a pretrained video backbone (Wan 2.2). Through zero-frame H2R and first-frame HI2R modes, Mitty achieves robust cross-task generalization and superior generation quality on Human2Robot and EPIC-Kitchens, outperforming rendering pipelines and generic video editors. The work provides a practical path toward scalable video-level supervision for robot learning and highlights the potential of end-to-end Human2Robot video generation as a precursor to full video-to-policy mappings.

Abstract

Learning directly from human demonstration videos is a key milestone toward scalable and generalizable robot learning. Yet existing methods rely on intermediate representations such as keypoints or trajectories, introducing information loss and cumulative errors that harm temporal and visual consistency. We present Mitty, a Diffusion Transformer that enables video In-Context Learning for end-to-end Human2Robot video generation. Built on a pretrained video diffusion model, Mitty leverages strong visual-temporal priors to translate human demonstrations into robot-execution videos without action labels or intermediate abstractions. Demonstration videos are compressed into condition tokens and fused with robot denoising tokens through bidirectional attention during diffusion. To mitigate paired-data scarcity, we also develop an automatic synthesis pipeline that produces high-quality human-robot pairs from large egocentric datasets. Experiments on Human2Robot and EPIC-Kitchens show that Mitty delivers state-of-the-art results, strong generalization to unseen environments, and new insights for scalable robot learning from human observations.

Figures (9)

• Figure 1: Overall architecture of Mitty. We build Mitty on a Diffusion Transformer–based video generation model and employ an In-Context Learning paradigm. The human demonstration video (input) and the noisy robot video latents (denoise stream) are concatenated, with noise injected only into the robot branch. A bidirectional attention mechanism enables cross-modal information flow, allowing the model to learn to generate robotic videos directly from human operation demonstrations.
• Figure 2: Starting from a human demonstration video, we first detect hands using Detectron2 wu2019detectron2 and then segment hands and arms using Segment Anything ravi2024sam. Next, we perform hand keypoint detection and inpaint the removed hand regions to obtain clean background frames. We then apply inverse kinematics solving to map the detected hand keypoints to robot arm poses and render the robot arms into the videos. Finally, with a human-in-the-loop filtering process, we curate over 6,000 high-quality synthetic human–robot paired videos to support the training of our Mitty model.
• Figure 3: Mitty’s generation results on Human2Robot and EPIC-Kitchens datasets. In each group of results, the first row shows the human demonstration videos, the second row shows the outputs generated by our method, and the third row shows the ground-truth robot execution videos.
• Figure 4: Masquerade’s multi-stage pipeline is prone to compounded errors (e.g., joint detection, inpainting, and rendering failures), as highlighted in red. In contrast, our curated training data enables a robust end-to-end model that produces more reliable Human2Robot mappings.
• Figure 5: Compared with state-of-the-art video editing models, the baseline methods take a robot reference image and a human demonstration video as input. However, even the most advanced baselines still struggle to maintain appearance and structural consistency of the robotic arm throughout the sequence.