EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World

TL;DR

EgoExoLearn addresses the challenge of bridging asynchronous procedural actions observed from egocentric and exocentric viewpoints by providing a large, multimodal dataset that pairs demonstration videos with corresponding egocentric recordings. The authors introduce four benchmarks—cross-view association, cross-view action anticipation & planning, cross-view referenced skill assessment, and cross-view captioning—along with gaze-enabled baselines to study how observations from one view map to actions in another. The dataset spans 120 hours across daily and laboratory tasks, includes fine-grained language annotations with timestamps, and provides calibrated gaze data to enable gaze-guided cross-view analysis. Overall, EgoExoLearn offers a valuable resource and empirical benchmarks that reveal current models' limitations in cross-view bridging and point toward future directions for embodied AI that can learn by observing humans in the real world.

Abstract

Being able to map the activities of others into one's own point of view is one fundamental human skill even from a very early age. Taking a step toward understanding this human ability, we introduce EgoExoLearn, a large-scale dataset that emulates the human demonstration following process, in which individuals record egocentric videos as they execute tasks guided by demonstration videos. Focusing on the potential applications in daily assistance and professional support, EgoExoLearn contains egocentric and demonstration video data spanning 120 hours captured in daily life scenarios and specialized laboratories. Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints. To this end, we present benchmarks such as cross-view association, cross-view action planning, and cross-view referenced skill assessment, along with detailed analysis. We expect EgoExoLearn can serve as an important resource for bridging the actions across views, thus paving the way for creating AI agents capable of seamlessly learning by observing humans in the real world. Code and data can be found at: https://github.com/OpenGVLab/EgoExoLearn

Figures (14)

• Figure 1: EgoExoLearn emulates the human asynchronous demonstration following process. It contains demonstration videos of multiple tasks, together with egocentric videos recorded by participants replicating the procedure after watching the demonstrations. The dataset comprises gaze signals and fine-grained multi-level multi-modal annotations, enabling the exploration of key features in this context such as cross-view association and cross-view action planning.
• Figure 2: The number of videos per task (left) and the average duration of each video per task (right). Task1 to Task5 represent the 5 daily tasks and the remaining are three tasks in specialized laboratories. In each recording session of the egocentric video, one participant may learn from multiple demonstration videos and one demonstration video may be watched by several participants.
• Figure 3: Occurrence and duration distribution of the annotated fine-level verbs and nouns associated with the right hand.
• Figure 4: Concept of the 3 benchmarks of cross-view association (Sec. \ref{['benchmark:association']}), cross-view action anticipation & planning (Sec. \ref{['benchmark:action']}) and cross-view reference skill assessment (Sec.\ref{['benchmark:skill']}) in this section. Other benchmarks can be found in the supplementary material.
• Figure S5: Cross-view association network with naive dual architecture (a) and improved architecture with additional gaze branch (b).