MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA

TL;DR

The paper tackles the scarcity of large-scale egocentric QA data and the challenge of long-form first-person video understanding. It introduces a narration-to-egocentric QA data engine to generate over 7 million QA pairs from Ego4D narrations, and the EgoMemoria benchmark (629 videos, 7,026 questions) with a debiasing evaluation protocol. The MM-Ego model leverages Memory Pointer Prompting to obtain a global video overview and selective high-resolution memory frames, enabling efficient reasoning over extended egocentric content. Empirical results show state-of-the-art mean debiased accuracy on EgoMemoria, strong performance on related benchmarks, and insightful analyses of frame usage and bias, underscoring the approach's potential for robust egocentric video understanding and memory-based QA.

Abstract

This research aims to comprehensively explore building a multimodal foundation model for egocentric video understanding. To achieve this goal, we work on three fronts. First, as there is a lack of QA data for egocentric video understanding, we automatically generate 7M high-quality QA samples for egocentric videos ranging from 30 seconds to one hour long in Ego4D based on human-annotated data. This is one of the largest egocentric QA datasets. Second, we contribute a challenging egocentric QA benchmark with 629 videos and 7,026 questions to evaluate the models' ability in recognizing and memorizing visual details across videos of varying lengths. We introduce a new de-biasing evaluation method to help mitigate the unavoidable language bias present in the models being evaluated. Third, we propose a specialized multimodal architecture featuring a novel "Memory Pointer Prompting" mechanism. This design includes a \textit{global glimpse} step to gain an overarching understanding of the entire video and identify key visual information, followed by a fallback step that utilizes the key visual information to generate responses. This enables the model to more effectively comprehend extended video content. With the data, benchmark, and model, we build MM-Ego, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding.

Figures (9)

• Figure 1: We introduce a foundation model for egocentric video understanding, contributing from three key perspectives: (a) 7 million egocentric QA samples generated from human narrations via a data engine, (b) a multimodal language model designed for egocentric video comprehension, and (c) the curation of a challenging egocentric video understanding benchmark.
• Figure 2: "Narration to Egocentric QA" data engine. Given a sequence of human-annotated video narrations, we instruct a language model (GPT-4o) to generate egocentric understanding-related questions and answers, along with identifying the key frames necessary to answer those questions.
• Figure 3: Video length distribution in our egocentric QA dataset.
• Figure 4: (a) Overview of the proposed Memory Pointer Prompting mechanism. Its inference consists of two steps: (1) Global Glimpse: We concatenate the compressed visual embeddings from all frames, denoted as $\mathbf{E}_{\text{vis}}^i$ for $i \in [1, N]$, with the question embeddings $\mathbf{E}_{\text{que}}^1$ and the memory pointer embedding $\mathbf{P}^1$. This combined embedding sequence is then input into the LLM. From the last layer, we extract embeddings and compute the dot product between the memory pointer embedding and all compressed visual embeddings to generate the correlation scores. The indices of the frames with the top $k$ scores are selected. During training, the correlation scores are supervised by ground-truth key frame indices via a binary cross-entropy loss. (2) Fallback: The high-resolution visual embeddings corresponding to the selected indices are fed into the LLM along with the question embeddings for final processing and response generation. (b) Illustration of LLM input sequence during training.
• Figure 5: The most frequently occurring verbs and nouns in EgoMemoria.