Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering

Yanjun Li, Yuqian Fu, Tianwen Qian, Qi'ao Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, Xiaoling Wang

TL;DR

This work introduces EgoCross, a comprehensive benchmark designed to evaluate the cross-domain generalization of MLLMs in EgocentricQA, and hopes EgoCross and the accompanying analysis will serve as a foundation for advancing domain-adaptive, robust egocentric video understanding.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly pushed the frontier of egocentric video question answering (EgocentricQA). However, existing benchmarks and studies are mainly limited to common daily activities such as cooking and cleaning. In contrast, real-world deployment inevitably encounters domain shifts, where target domains differ substantially in both visual style and semantic content. To bridge this gap, we introduce \textbf{EgoCross}, a comprehensive benchmark designed to evaluate the cross-domain generalization of MLLMs in EgocentricQA. EgoCross covers four diverse and challenging domains, including surgery, industry, extreme sports, and animal perspective, representing realistic and high-impact application scenarios. It comprises approximately 1,000 QA pairs across 798 video clips, spanning four key QA tasks: prediction, recognition, localization, and counting. Each QA pair provides both OpenQA and CloseQA formats to support fine-grained evaluation. Extensive experiments show that most existing MLLMs, whether general-purpose or egocentric-specialized, struggle to generalize to domains beyond daily life, highlighting the limitations of current models. Furthermore, we conduct several pilot studies, e.g., fine-tuning and reinforcement learning, to explore potential improvements. We hope EgoCross and our accompanying analysis will serve as a foundation for advancing domain-adaptive, robust egocentric video understanding.

EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering

TL;DR

This work introduces EgoCross, a comprehensive benchmark designed to evaluate the cross-domain generalization of MLLMs in EgocentricQA, and hopes EgoCross and the accompanying analysis will serve as a foundation for advancing domain-adaptive, robust egocentric video understanding.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly pushed the frontier of egocentric video question answering (EgocentricQA). However, existing benchmarks and studies are mainly limited to common daily activities such as cooking and cleaning. In contrast, real-world deployment inevitably encounters domain shifts, where target domains differ substantially in both visual style and semantic content. To bridge this gap, we introduce \textbf{EgoCross}, a comprehensive benchmark designed to evaluate the cross-domain generalization of MLLMs in EgocentricQA. EgoCross covers four diverse and challenging domains, including surgery, industry, extreme sports, and animal perspective, representing realistic and high-impact application scenarios. It comprises approximately 1,000 QA pairs across 798 video clips, spanning four key QA tasks: prediction, recognition, localization, and counting. Each QA pair provides both OpenQA and CloseQA formats to support fine-grained evaluation. Extensive experiments show that most existing MLLMs, whether general-purpose or egocentric-specialized, struggle to generalize to domains beyond daily life, highlighting the limitations of current models. Furthermore, we conduct several pilot studies, e.g., fine-tuning and reinforcement learning, to explore potential improvements. We hope EgoCross and our accompanying analysis will serve as a foundation for advancing domain-adaptive, robust egocentric video understanding.

Paper Structure

This paper contains 44 sections, 16 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Egocentric Video Understanding
  4. MLLMs for Video Understanding
  5. Cross-Domain Generalization
  6. EgoCross Dataset
  7. Source Selection and Task Taxonomy
  8. Design Principles.
  9. Domain and Data Source Selection.
  10. QA Task Taxonomy.
  11. Data Curation Pipeline
  12. Meta Annotation Refinement.
  13. QA Template Design.
  14. Batch Generation and Final Quality Control.
  15. Dataset Statistics
  16. ...and 29 more sections

Figures (16)

  • Figure 1: Examples of Our EgoCross Benchmark. We go beyond everyday egocentric scenarios, covering four diverse, cross-domain, application-oriented areas: Surgery, Industry, Extreme Sports, and Animal Perspective. As shown in the examples, both the visual appearances and the semantic content differ significantly from existing EgocentricQA datasets.
  • Figure 2: Data construction pipeline of EgoCross.
  • Figure 3: An overview of the EgoCross task taxonomy and statistics. (Top-left) The overall distribution of the four main task categories: Identification, Localization, Prediction, and Counting. (Top-right) The number of questions across the four primary domains. (Bottom) A selection of representative QA examples for each major capability is presented. For a more comprehensive list of examples, please see the Appendix.
  • Figure 4: t-SNE visualization of text and visual features. EgoCross domains are color-coded: Surgery (red), Industry (blue), ExtremeSports (green), Animal Perspective (orange) and Daily-activity (purple).
  • Figure 5: In-domain and cross-domain accuracy comparison across five QA types: Action Temporal Localization (ATL); Next Action Prediction (NAP); Interaction Identification (II); Special Action Identification (SAI); Action Sequence Identification (ASI). The results highlight the performance gap when evaluating on novel domains.
  • ...and 11 more figures