Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Egocentric Video-Language Pretraining

Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, Mike Zheng Shou

TL;DR

This work addresses the gap in egocentric video-language pretraining by introducing EgoClip, a large-scale 1st-person pretraining dataset derived from Ego4D, and EgoNCE, an egocentric-aware contrastive objective that leverages action and scene context. The authors also propose EgoMCQ, a development benchmark tightly aligned with EgoClip to facilitate rapid iteration. Their approach, based on a Frozen dual-encoder architecture with TimeSformer and DistillBERT, achieves state-of-the-art transfer across five egocentric downstream tasks, including video-text retrieval, action recognition, and moment/ language-grounding challenges. This work lays a foundation for scalable, domain-specific VLP in egocentric vision, with implications for AR, robotics, and daily-life AI systems.

Abstract

Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.

Egocentric Video-Language Pretraining

TL;DR

This work addresses the gap in egocentric video-language pretraining by introducing EgoClip, a large-scale 1st-person pretraining dataset derived from Ego4D, and EgoNCE, an egocentric-aware contrastive objective that leverages action and scene context. The authors also propose EgoMCQ, a development benchmark tightly aligned with EgoClip to facilitate rapid iteration. Their approach, based on a Frozen dual-encoder architecture with TimeSformer and DistillBERT, achieves state-of-the-art transfer across five egocentric downstream tasks, including video-text retrieval, action recognition, and moment/ language-grounding challenges. This work lays a foundation for scalable, domain-specific VLP in egocentric vision, with implications for AR, robotics, and daily-life AI systems.

Abstract

Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.
Paper Structure (38 sections, 6 equations, 13 figures, 12 tables)

This paper contains 38 sections, 6 equations, 13 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. EgoClip: An Egocentric Video-Language Pretraining Dataset
  4. Video-Language Pretraining Model
  5. Architecture: Dual-encoder Pipeline
  6. EgoNCE: An Egocentric-friendly Pretraining Objective
  7. EgoMCQ: A Benchmark for Egocentric VLP Development
  8. Experiments
  9. Benchmarks and Settings
  10. Ablation Studies
  11. Ablation of the strategy used when creating EgoClip.
  12. Comparisons with State-of-the-arts
  13. Conclusion, Limitations, and Societal Impacts.
  14. Acknowledgements
  15. Differentiating Egocentric VLP and Ego4D
  16. ...and 23 more sections

Figures (13)

  • Figure 1: Our Egocentric VLP includes: (a) the pretraining set EgoClip, (b) the VLP model, and (c) the development set EgoMCQ. We use EgoClip to pretrain a VLP model with the EgoNCE loss and then evaluate on EgoMCQ. According to the feedback, we iteratively refine our designs of (a) and (b). We then transfer the pretrained model to downstream tasks relevant to the egocentric domain.
  • Figure 2: Design of the Egocentric VLP development set. Top: An illustration of why the task of text-video retrieval is not suitable; Bottom: Two settings of EgoMCQ. Left-bottom: The "inter-video" setting, each question contains $5$ clips from different videos. Right-bottom: The "intra-video" setting, each question contains $5$ contiguous clips from the same video, making it more challenging.
  • Figure 3: Institution distribution of EgoClip
  • Figure 4: Scenario distribution of EgoClip
  • Figure 5: Scenario distribution of EgoMCQ
  • ...and 8 more figures