EgoMI: Learning Active Vision and Whole-Body Manipulation from Egocentric Human Demonstrations

TL;DR

EgoMI addresses the embodiment gap in imitation learning by capturing synchronized egocentric head and hand trajectories and enabling whole-body retargeting to semi-humanoid robots. It introduces SPARKS, a memory-based keyframe selector, and a two-stage finetuning pipeline to map 29D action/state space to relative Cartesian space for robust zero-shot transfer. Real-world experiments on tabletop and shelf tasks demonstrate that incorporating head motion and SPARKS memory improves success rates and enables long-horizon manipulation without on-robot demonstrations. The approach offers a scalable path to bridging human-robot differences in perception and embodiment, enabling robust imitation learning with minimal domain adaptation or visual augmentation.

Abstract

Imitation learning from human demonstrations offers a promising approach for robot skill acquisition, but egocentric human data introduces fundamental challenges due to the embodiment gap. During manipulation, humans actively coordinate head and hand movements, continuously reposition their viewpoint and use pre-action visual fixation search strategies to locate relevant objects. These behaviors create dynamic, task-driven head motions that static robot sensing systems cannot replicate, leading to a significant distribution shift that degrades policy performance. We present EgoMI (Egocentric Manipulation Interface), a framework that captures synchronized end-effector and active head trajectories during manipulation tasks, resulting in data that can be retargeted to compatible semi-humanoid robot embodiments. To handle rapid and wide-spanning head viewpoint changes, we introduce a memory-augmented policy that selectively incorporates historical observations. We evaluate our approach on a bimanual robot equipped with an actuated camera head and find that policies with explicit head-motion modeling consistently outperform baseline methods. Results suggest that coordinated hand-eye learning with EgoMI effectively bridges the human-robot embodiment gap for robust imitation learning on semi-humanoid embodiments. Project page: https://egocentric-manipulation-interface.github.io

Figures (6)

• Figure 1: Overview of the EgoMI framework. EgoMI captures egocentric human demonstrations with synchronized head and hand tracking. To handle rapid viewpoint changes from head motion, demonstrations are processed via Spatial-Aware Robust Keyframe Selection (SPARKS). A two-stage fine-tuning procedure then adapts a pre-trained absolute joint-space foundation model ($\pi_0$) into Relative-Operation Space. The learned policy remarkably transfers zero-shot to real robots through whole-body retargeting, without requiring any visual augmentation, explicit visual alignment, or on-embodiment data collection.
• Figure 2: EgoMI policy deployment setup. We use a modified Rainbow RBY1 robot with a 6-DoF YAM i2rt_yam_arm + ZED2i stereolabs_zed2i camera mounted on top as the fully actuated head. The gripper configuration is identical to the human demonstration setup, minimizing the embodiment gap.
• Figure 3: Tabletop Task Rollout Sequence: (Left). The images show a real 29D policy evaluation rollout where the robot (1) scans for target cans across a cluttered workspace, (2) grasps the correct item with potential handoff between grippers, and (3) places it into the designated bin. (Right). The Sankey diagrams illustrate failure modes between policies with full 29D action space and active head-camera versus reduced 20D wrist camera-only and 20D + head-camera images baselines.
• Figure 4: Randomization distribution and example initial configuration of the tabletop environment highlighting the wide distribution of object positions and clutter scenarios used during evaluation rollouts. (Right). Initial configurations for target object may be outside of the immediate field of view of the initialized robot during experimentation. Target object and placement location may also reside on opposite ends of the workspace requiring a bi-manual handoff maneuver.
• Figure 5: Shelf Task Rollout Sequence. (Left). The images show a real 29D policy evaluation rollout where the robot (1) scans across multiple shelf tiers to locate target cans, (2) reaches and grasps the selected item, (3) performs a mid-air inter-gripper handoff, and (4) places the item into a shopping basket, then repeats on the remaining can. (Right). The Sankey diagrams show failure modes for the 29D active-head, whole-body retargeted policy compared to the 20D wrist camera-only policy.