ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos

TL;DR

ZeroMimic is carefully designed to exploit recent advances in semantic and geometric visual understanding of human videos, together with modern grasp affordance detectors and imitation policy classes, and to enable plug-and-play reuse of ZeroMimic policies on other task setups and robots.

Abstract

Many recent advances in robotic manipulation have come through imitation learning, yet these rely largely on mimicking a particularly hard-to-acquire form of demonstrations: those collected on the same robot in the same room with the same objects as the trained policy must handle at test time. In contrast, large pre-recorded human video datasets demonstrating manipulation skills in-the-wild already exist, which contain valuable information for robots. Is it possible to distill a repository of useful robotic skill policies out of such data without any additional requirements on robot-specific demonstrations or exploration? We present the first such system ZeroMimic, that generates immediately deployable image goal-conditioned skill policies for several common categories of manipulation tasks (opening, closing, pouring, pick&place, cutting, and stirring) each capable of acting upon diverse objects and across diverse unseen task setups. ZeroMimic is carefully designed to exploit recent advances in semantic and geometric visual understanding of human videos, together with modern grasp affordance detectors and imitation policy classes. After training ZeroMimic on the popular EpicKitchens dataset of ego-centric human videos, we evaluate its out-of-the-box performance in varied real-world and simulated kitchen settings with two different robot embodiments, demonstrating its impressive abilities to handle these varied tasks. To enable plug-and-play reuse of ZeroMimic policies on other task setups and robots, we release software and policy checkpoints of our skill policies.

Figures (13)

• Figure 1: ZeroMimic distills robotic manipulation skills from egocentric web videos for zero-shot deployment [id=js]across diverse real-world and simulated environments, a variety of objects, and different robot embodiments.
• Figure 2: Representative related work organized by Generality of Source Human Videos and Level of Knowledge Transfer. ZeroMimic learns diverse zero-shot policies from in-the-wild web videos.
• Figure 3: ZeroMimic is composed of the grasping [id=js]phase and the post-grasp [id=js]phase. The grasping phase (top) [id=js]leverages human affordance-based grasping to execute a task-relevant grasp. The post-grasp phase (bottom) is an imitation policy trained on web videos to predict 6D wrist trajectories. We deploy this trained model directly on the robot.
• Figure 4: 6D wrist post-grasp policy outputs on unseen images. The red, green, and blue arrows denote the $x, y, z$ coordinates of the wrist orientation in the camera frame.
• Figure 5: ZeroMimic Zero-Shot Performance Overview. ZeroMimic demonstrates strong generalization capabilities, achieving consistent success across diverse tasks, robot embodiments, and both real-world and simulated environments. The evaluation spans 34 distinct scenarios across 18 object categories in 7 kitchen scenes, highlighting the adaptability and robustness of the system. For a detailed breakdown of performance by skills, robots, object categories, and scenarios, refer to Appendix \ref{['app:full_results']}.