Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learning

TL;DR

This work introduces Motion Track Policy (MT-pi), an imitation learning framework that learns from predominantly human video by predicting short-horizon 2D image-space motion tracks as actions, enabling cross-embodiment transfer to robots. A Keypoint Retargeting Network and a diffusion-based motion-predictor are trained on a hybrid dataset of human and robot demonstrations, with auxiliary losses to align visual embeddings across embodiments. At test time, stereo 2D tracks are triangulated to yield 6DoF robot actions, achieving an average of 86.5% success over four real-world tasks and outperforming baselines that do not leverage human video or this action space. The approach significantly reduces robot-data requirements while enabling generalization to human-video-only scenarios, and the authors provide open-source code and videos for reproducible research.

Abstract

Teaching robots to autonomously complete everyday tasks remains a challenge. Imitation Learning (IL) is a powerful approach that imbues robots with skills via demonstrations, but is limited by the labor-intensive process of collecting teleoperated robot data. Human videos offer a scalable alternative, but it remains difficult to directly train IL policies from them due to the lack of robot action labels. To address this, we propose to represent actions as short-horizon 2D trajectories on an image. These actions, or motion tracks, capture the predicted direction of motion for either human hands or robot end-effectors. We instantiate an IL policy called Motion Track Policy (MT-pi) which receives image observations and outputs motion tracks as actions. By leveraging this unified, cross-embodiment action space, MT-pi completes tasks with high success given just minutes of human video and limited additional robot demonstrations. At test time, we predict motion tracks from two camera views, recovering 6DoF trajectories via multi-view synthesis. MT-pi achieves an average success rate of 86.5% across 4 real-world tasks, outperforming state-of-the-art IL baselines which do not leverage human data or our action space by 40%, and generalizes to scenarios seen only in human videos. Code and videos are available on our website https://portal-cornell.github.io/motion_track_policy/.

Figures (5)

• Figure 1: Motion Track Policy (MT-$\pi$ Overview): Left: Using 10 minutes of human video and a small set of robot demonstrations, we train MT-$\pi$ to take a third-person image observation ($I_t$) and output motion tracks ($\hat{s}$), a cross-embodiment action space of 2D trajectories in image space representing manipulator movement. Right: At test-time, we predict $\hat{s}$ along with grasps ($\hat{g}$) from 2 camera views (1 shown) and recover full 6DoF robot actions using stereo triangulation for execution.
• Figure 2: MT-$\pi$ Policy Architecture. We co-train MT-$\pi$ on human and robot demonstrations to predict the future pixel locations of keypoints on the end-effector (shown in red). For robot demonstrations, keypoints are extracted using calibrated camera-to-robot extrinsics, while human hand keypoints are obtained via HaMeR pavlakos2024reconstructing. To address embodiment differences, a Keypoint Retargeting Network maps robot keypoints to more closely resemble the human hand structure. The Motion Prediction Network, based on Diffusion Policy, takes image embeddings and current keypoints as input and predicts future keypoint tracks and grasp states. By operating entirely in image space, MT-$\pi$ directly learns actions from both robot and human demonstrations with a cross-embodiment action representation.
• Figure 3: MT-$\pi$ Action Inference: MT-$\pi$ represent actions as 2D image trajectories which are not directly executable on a robot. To bridge this, we predict motion tracks from two third-person camera views and treat them as pixelwise correspondences. Using stereo triangulation with known extrinsics, we recover 3D keypoints and compute the rigid transformation between consecutive timesteps. This yields a 6DoF trajectory, $a_{t:t+T}$, for robot execution. In practice, we use a much shorter prediction horizon ($H=16 \ll T$) for more closed-loop reasoning.
• Figure 4: Effect of Data Distribution on Policy Success: We evaluate MT-$\pi$ against DP and ACT on the Serve Egg task (see \ref{['fig:front']}), measuring policy success (green = high, red = low) subject to varying amounts of robot and human training data. While all policies improve with more robot data, collecting teleoperated demonstrations takes nearly 5x longer than human videos (70 human demos $\sim$ 10 mins., 100 robot demos $\sim$ 65 mins.). MT-$\pi$ achieves strong performance even in the low robot data regime by leveraging just $5-10$ minutes of human video, suggesting further gains are possible by fixing the amount of robot data and scaling human data alone.
• Figure 5: Generalization to Motions Seen in Human-Video Only: We evaluate two variants of MT-$\pi$ (trained on human + robot data vs. robot data only) against DP and ACT for the task of closing a drawer. Human videos include closing the drawer in both directions, while robot demonstrations only show closing to the right. While all policies perform well closing to the right (in-distribution for $D_\mathrm{robot}$), only MT-$\pi$ (H+R) generalizes to closing the drawer to the left.