Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding

Wenqi Zhou, Kai Cao, Hao Zheng, Yunze Liu, Xinyi Zheng, Miao Liu, Per Ola Kristensson, Walterio Mayol-Cuevas, Fan Zhang, Weizhe Lin, Junxiao Shen

TL;DR

The paper introduces X-LeBench, the first benchmark for ultra-long egocentric video understanding, addressing a gap where existing datasets focus on short clips and fail to capture long-term daily activities from a first-person perspective. It presents a life-logging simulation pipeline that combines synthetic daily plans with real Ego4D footage to produce 432 coherent life logs spanning 23 minutes to 16.4 hours, enabling evaluation of long-horizon tasks. Through experiments with Gemini-1.5-Flash, Socratic Models, Retrieve-Socratic, and LongVU, the study reveals that temporal localization, memory integration, and long-context reasoning remain major bottlenecks, while textual representations via LLMs offer scalable gains. The work demonstrates the value of a controllable, multi-hour benchmark for advancing temporally aware AI systems capable of long-term memory, planning, and daily-life understanding, and highlights directions for improving long-form video reasoning and retrieval-based architectures.

Abstract

Long-form egocentric video understanding provides rich contextual information and unique insights into long-term human behaviors, holding significant potential for applications in embodied intelligence, long-term activity analysis, and personalized assistive technologies. However, existing benchmark datasets primarily focus on single, short (\eg, minutes to tens of minutes) to moderately long videos, leaving a substantial gap in evaluating extensive, ultra-long egocentric video recordings. To address this, we introduce X-LeBench, a novel benchmark dataset meticulously designed to fill this gap by focusing on tasks requiring a comprehensive understanding of extremely long egocentric video recordings. Our X-LeBench develops a life-logging simulation pipeline that produces realistic, coherent daily plans aligned with real-world video data. This approach enables the flexible integration of synthetic daily plans with real-world footage from Ego4D-a massive-scale egocentric video dataset covers a wide range of daily life scenarios-resulting in 432 simulated video life logs spanning from 23 minutes to 16.4 hours. The evaluations of several baseline systems and multimodal large language models (MLLMs) reveal their poor performance across the board, highlighting the inherent challenges of long-form egocentric video understanding, such as temporal localization and reasoning, context aggregation, and memory retention, and underscoring the need for more advanced models.

X-LeBench: A Benchmark for Extremely Long Egocentric Video Understanding

TL;DR

The paper introduces X-LeBench, the first benchmark for ultra-long egocentric video understanding, addressing a gap where existing datasets focus on short clips and fail to capture long-term daily activities from a first-person perspective. It presents a life-logging simulation pipeline that combines synthetic daily plans with real Ego4D footage to produce 432 coherent life logs spanning 23 minutes to 16.4 hours, enabling evaluation of long-horizon tasks. Through experiments with Gemini-1.5-Flash, Socratic Models, Retrieve-Socratic, and LongVU, the study reveals that temporal localization, memory integration, and long-context reasoning remain major bottlenecks, while textual representations via LLMs offer scalable gains. The work demonstrates the value of a controllable, multi-hour benchmark for advancing temporally aware AI systems capable of long-term memory, planning, and daily-life understanding, and highlights directions for improving long-form video reasoning and retrieval-based architectures.

Abstract

Long-form egocentric video understanding provides rich contextual information and unique insights into long-term human behaviors, holding significant potential for applications in embodied intelligence, long-term activity analysis, and personalized assistive technologies. However, existing benchmark datasets primarily focus on single, short (\eg, minutes to tens of minutes) to moderately long videos, leaving a substantial gap in evaluating extensive, ultra-long egocentric video recordings. To address this, we introduce X-LeBench, a novel benchmark dataset meticulously designed to fill this gap by focusing on tasks requiring a comprehensive understanding of extremely long egocentric video recordings. Our X-LeBench develops a life-logging simulation pipeline that produces realistic, coherent daily plans aligned with real-world video data. This approach enables the flexible integration of synthetic daily plans with real-world footage from Ego4D-a massive-scale egocentric video dataset covers a wide range of daily life scenarios-resulting in 432 simulated video life logs spanning from 23 minutes to 16.4 hours. The evaluations of several baseline systems and multimodal large language models (MLLMs) reveal their poor performance across the board, highlighting the inherent challenges of long-form egocentric video understanding, such as temporal localization and reasoning, context aggregation, and memory retention, and underscoring the need for more advanced models.
Paper Structure (31 sections, 1 equation, 6 figures, 9 tables)

This paper contains 31 sections, 1 equation, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Egocentric Video Benchmarks
  4. Long-form Video Benchmarks
  5. LLM-assisted Annotation Scheme
  6. X-LeBench
  7. Life-logging Simulation Pipeline
  8. Benchmark Tasks
  9. Experiments
  10. Settings
  11. Results
  12. Summary of Key Findings
  13. Conclusion
  14. Limitations
  15. Persona Profile
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Example of generated video life logs in X-LeBench. Generated video life logs consist of multiple videos with corresponding timestamps. The visualization shows the time organization and content allocation of data.
  • Figure 2: Overview of life-logging simulation pipeline. Stage 1: Generation of personalized persona profiles and daily plan chunks based on predefined parameters. Stage 2: Core information (time, scene, content) extraction of videos in the video selection library. Stage 3: Matching and retrieval of daily plan chunks with videos, and life-log simulations are iteratively refined through reflection, resulting in the final optimized output.
  • Figure 3: Dataset statistics. (a) Video selection library consists of 7852 videos from Ego4D dataset, covering 135 scenarios, here shows the top 30. (b) X-LeBench has video life-log durations spanning from 23 minutes to 16.4 hours, here shows distribution of duration range across different data categories. (c) and (d) show the distribution of scene and time information categories of videos in video selection library, respectively. (e) Statistical information on duration lengths of different data categories, including minimum, mean, median and maximum values. (f) The distribution of number of videos with different occurrences.
  • Figure 4: Performance comparison on X-LeBench. (a): Radar chart showing the overall performance of each method across all tasks. For fair comparison across tasks, scores for summarization are scaled by a factor of 10 (maximum 100). (b): The performance comparison across different data categories for different tasks.
  • Figure 5: Example of input context.
  • ...and 1 more figures