Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos
X. Hu, G. Ye
TL;DR
The paper tackles the data collection bottleneck for robotic manipulation by exploiting Internet videos and overcoming limitations of prior methods that undervalue temporal coherence and key entities. It introduces a two-stage framework that first uses vision-language foundation models to propose semantic keypoints and then applies dense trajectory tracking to recover full manipulation sequences. The approach yields temporally coherent, multi-keypoint trajectories for hands, tools, and objects, enabling scalable, low-cost data for pretraining robotic policies. By mapping human motion to robot end-effectors, it supports cross-domain transfer and broader applicability across robotic platforms.
Abstract
Collecting high-quality data for training large-scale robotic models typically relies on real robot platforms, which is labor-intensive and costly, whether via teleoperation or scripted demonstrations. To scale data collection, many researchers have turned to leveraging human manipulation videos available online. However, current methods predominantly focus on hand detection or object pose estimation, failing to fully exploit the rich interaction cues embedded in these videos. In this work, we propose a novel approach that combines large foundation models for video understanding with point tracking techniques to extract dense trajectories of all task-relevant keypoints during manipulation. This enables more comprehensive utilization of Internet-scale human demonstration videos. Experimental results demonstrate that our method can accurately track keypoints throughout the entire manipulation process, paving the way for more scalable and data-efficient robot learning.