EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World

TL;DR

EgoMe introduces a large-scale egocentric dataset that records imitator-centered data paired with exocentric observations to model human imitation learning in real-world settings. It enriches the data with eye gaze and IMU signals, plus coarse and fine language annotations and imitative verification, enabling cross-view analyses. The work defines a comprehensive benchmark suite—covering exo-ego retrieval, gaze prediction, imitative assessment, reorganization, and both coarse and fine-level understanding—that demonstrates the dataset's potential to advance embodied intelligence and robot learning from observation to imitation. By providing strong baselines and a public resource, EgoMe aims to bridge gaps between exocentric perception, egocentric action, and cross-view imitation understanding with practical implications for real-world robotics and AR/VR systems.

Abstract

In human imitation learning, the imitator typically take the egocentric view as a benchmark, naturally transferring behaviors observed from an exocentric view to their owns, which provides inspiration for researching how robots can more effectively imitate human behavior. However, current research primarily focuses on the basic alignment issues of ego-exo data from different cameras, rather than collecting data from the imitator's perspective, which is inconsistent with the high-level cognitive process. To advance this research, we introduce a novel large-scale egocentric dataset, called EgoMe, which towards following the process of human imitation learning via the imitator's egocentric view in the real world. Our dataset includes 7902 paired exo-ego videos (totaling15804 videos) spanning diverse daily behaviors in various real-world scenarios. For each video pair, one video captures an exocentric view of the imitator observing the demonstrator's actions, while the other captures an egocentric view of the imitator subsequently following those actions. Notably, EgoMe uniquely incorporates exo-ego eye gaze, other multi-modal sensor IMU data and different-level annotations for assisting in establishing correlations between observing and imitating process. We further provide a suit of challenging benchmarks for fully leveraging this data resource and promoting the robot imitation learning research. Extensive analysis demonstrates significant advantages over existing datasets. Our EgoMe dataset and benchmarks are available at https://huggingface.co/datasets/HeqianQiu/EgoMe.

Figures (10)

• Figure 1: Our EgoMe dataset takes the egocentric view as a benchmark, following behaviors observed from a exocentric view to their own actions for exploring the process of human imitation learning in the real world.
• Figure 2: Illustrative example in our EgoMe dataset. It includes eye gaze, action label, coarse/fine-level language annotations, and imitative verification annotation during the observing (exocentric) and following (egocentric) process.
• Figure 3: Statistics of the proposed EgoMe dataset.
• Figure 4: Overview of all baseline models for the proposed benchmarks.
• Figure 5: Statistic of exocentric/egocentric video duration and correct/incorrect following video pairs.