Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation

Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, Yu Qiao

TL;DR

This work introduces EgoVideo, an egocentric video foundation model built via a three-stage pipeline: augmented data selection from diverse egocentric sources, post-training on a strong video-language backbone, and downstream task-specific adaptation. By leveraging InternVideo2 as the base and a visual-text contrastive objective, EgoVideo-V and EgoVideo-T are tuned for egocentric understanding and then finetuned across eight Ego4D/EPIC-Kitchens tasks, including NLQ, Step Grounding, Moment Queries, object-interaction anticipation, long-term action anticipation, action recognition, multi-instance retrieval, and domain adaptation. The results demonstrate substantial gains over baselines and state-of-the-art on multiple tracks, with Vicuna-7B enhancing long-term action anticipation and strong EK100 performance in action recognition and retrieval. The work highlights the feasibility and benefits of a unified egocentric foundation model for diverse first-person vision tasks, and provides public code and pretrained models for reproducibility and further research.

Abstract

In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the unique characteristics of egocentric videos and provides strong support for our competition submissions. In the Ego4D challenges, we tackle various tasks including Natural Language Queries, Step Grounding, Moment Queries, Short-term Object Interaction Anticipation, and Long-term Action Anticipation. In addition, we also participate in the EPIC-Kitchens challenge, where we engage in the Action Recognition, Multiple Instance Retrieval, and Domain Adaptation for Action Recognition tracks. By adapting EgoVideo to these diverse tasks, we showcase its versatility and effectiveness in different egocentric video analysis scenarios, demonstrating the powerful representation ability of EgoVideo as an egocentric foundation model. Our codebase and pretrained models are publicly available at https://github.com/OpenGVLab/EgoVideo.

EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation

TL;DR

This work introduces EgoVideo, an egocentric video foundation model built via a three-stage pipeline: augmented data selection from diverse egocentric sources, post-training on a strong video-language backbone, and downstream task-specific adaptation. By leveraging InternVideo2 as the base and a visual-text contrastive objective, EgoVideo-V and EgoVideo-T are tuned for egocentric understanding and then finetuned across eight Ego4D/EPIC-Kitchens tasks, including NLQ, Step Grounding, Moment Queries, object-interaction anticipation, long-term action anticipation, action recognition, multi-instance retrieval, and domain adaptation. The results demonstrate substantial gains over baselines and state-of-the-art on multiple tracks, with Vicuna-7B enhancing long-term action anticipation and strong EK100 performance in action recognition and retrieval. The work highlights the feasibility and benefits of a unified egocentric foundation model for diverse first-person vision tasks, and provides public code and pretrained models for reproducibility and further research.

Abstract

In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the unique characteristics of egocentric videos and provides strong support for our competition submissions. In the Ego4D challenges, we tackle various tasks including Natural Language Queries, Step Grounding, Moment Queries, Short-term Object Interaction Anticipation, and Long-term Action Anticipation. In addition, we also participate in the EPIC-Kitchens challenge, where we engage in the Action Recognition, Multiple Instance Retrieval, and Domain Adaptation for Action Recognition tracks. By adapting EgoVideo to these diverse tasks, we showcase its versatility and effectiveness in different egocentric video analysis scenarios, demonstrating the powerful representation ability of EgoVideo as an egocentric foundation model. Our codebase and pretrained models are publicly available at https://github.com/OpenGVLab/EgoVideo.
Paper Structure (15 sections, 1 figure, 10 tables)

This paper contains 15 sections, 1 figure, 10 tables.

Table of Contents

  1. Introduction
  2. Training Process of EgoVideo
  3. Stage1: Augmented Data Selection
  4. Stage2: Egocentric Video Post-training
  5. Stage 3: Egocentric Downstream Adaptation
  6. Task-specific Finetuning
  7. Task 1: Natural Language Queries @ Ego4D
  8. Task 2: GoalStep - Step Grounding @ Ego4D
  9. Task 3: Moment Queries @ Ego4D
  10. Task 4: Short-term Object-interaction Anticipation @ Ego4D
  11. Task 5: Long-term Action Anticipation @ Ego4D
  12. Task6: Action Recognition @ EPIC
  13. Task7: Multi-instance Retrieval @ EPIC
  14. Task8: Domain Adaptation for Action Recognition @ EPIC
  15. Limitation and Conclusion

Figures (1)

  • Figure 1: The workflow of the training process of EgoVideo. It includes 3 stages: in the first stage, we filter and select high-quality egocentric video and text pairs from multiple existing datasets. Then we perform post pertaining using the data in stage 1 by standard video-text contrastive learning. Finally, we adapt the pretrained EgoVideo model to different downstream tasks.