EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation

TL;DR

This work presents EgoVid-5M, a large-scale egocentric video generation dataset comprising 5 million 1080p clips with fine-grained kinematic and textual action annotations, coupled with a dedicated data cleaning pipeline to ensure frame coherence and realistic motion. It introduces EgoDreamer, a generator driven by both high-level action descriptions and low-level kinematic signals, facilitated by a Unified Action Encoder and Adaptive Alignment for multi-scale control. Through extensive experiments, EgoVid-5M improves baseline egocentric video generation across multiple metrics, and EgoDreamer demonstrates precise, controllable video synthesis, validated by diverse ablations and comprehensive evaluation. The dataset, its annotations, and cleaning metadata are released to advance research in action-driven egocentric video generation for VR/AR and gaming applications.

Abstract

Video generation has emerged as a promising tool for world simulation, leveraging visual data to replicate real-world environments. Within this context, egocentric video generation, which centers on the human perspective, holds significant potential for enhancing applications in virtual reality, augmented reality, and gaming. However, the generation of egocentric videos presents substantial challenges due to the dynamic nature of egocentric viewpoints, the intricate diversity of actions, and the complex variety of scenes encountered. Existing datasets are inadequate for addressing these challenges effectively. To bridge this gap, we present EgoVid-5M, the first high-quality dataset specifically curated for egocentric video generation. EgoVid-5M encompasses 5 million egocentric video clips and is enriched with detailed action annotations, including fine-grained kinematic control and high-level textual descriptions. To ensure the integrity and usability of the dataset, we implement a sophisticated data cleaning pipeline designed to maintain frame consistency, action coherence, and motion smoothness under egocentric conditions. Furthermore, we introduce EgoDreamer, which is capable of generating egocentric videos driven simultaneously by action descriptions and kinematic control signals. The EgoVid-5M dataset, associated action annotations, and all data cleansing metadata will be released for the advancement of research in egocentric video generation.

Figures (15)

• Figure 1: EgoVid-5M is a meticulously curated high-quality action-video dataset designed specifically for egocentric video generation. It includes detailed action annotations, such as fine-grained kinematic control and high-level textual descriptions. Furthermore, it incorporates robust data cleaning strategies to ensure frame consistency, action coherence, and motion smoothness under egocentric conditions.
• Figure 2: Data annotation pipeline and cleansing metadata of EgoVid-5M.
• Figure 3: Data annotation distribution of EgoVid-5M. (a) and (b) describe the quantities of the top 20 verbs and nouns. (c) Text-video action alignment is assessed using the EgoVideo score. (d) and (e) measure the semantic similarity between text and frames and between frames and the first frame using the average CLIP score. (f) Motion smoothness is quantified by the variance of translation and rotation. (g) Motion strength is represented by the average global optical flow. (h) Video clarity is determined by the DOVER score.
• Figure 4: The overall framework of EgoDreamer. EgoDreamer introduces (a) the Unified Action Encoder to embed different action inputs simultaneously, and it utilizes (b) the Adaptive Alignment to integrate action conditions into the egocentric video generation branch (c).
• Figure 5: The video visualization comparison across different data cleaning strategies reveals distinct outcomes, where the blue box highlights the difference. Videos generated by strategy-1 fail to capture local motion and tend to be stationary. In contrast, videos produced by strategy-2 exhibit excessive motion, compromising semantic coherence. Meanwhile, videos generated by strategy-3 effectively model intricate hand movements, striking a balance between motion strength and semantic fidelity.