Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ego-Grounding for Personalized Question-Answering in Egocentric Videos

Junbin Xiao, Shenglang Zhang, Pengxiang Zhu, Angela Yao

Abstract

We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo

Ego-Grounding for Personalized Question-Answering in Egocentric Videos

Abstract

We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo

Paper Structure

This paper contains 19 sections, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. MyEgo Dataset
  4. Task Definition
  5. Dataset Construction
  6. Dataset Analysis
  7. Dataset Statistics
  8. Dataset Comparison
  9. Experiments
  10. Experimental Setups
  11. Main Results
  12. Controlled Analyses
  13. Conclusion
  14. MyEgo Dataset
  15. Video Source Introduction
  16. ...and 4 more sections

Figures (9)

  • Figure 1: Example of egocentric personalized QA concerning my demands. The narratives are provided for better understanding.
  • Figure 2: MyEgo examples. To answer my questions, the models must understand (a) which hand is mine, (b) memorize and track the green chess pieces I used (to be distinguished with the red ones of another person, and the blue ones which are not used).
  • Figure 3: Statistic analysis of MyEgo
  • Figure 4: Result visualization.
  • Figure 5: Accuracy distribution along the video stream. Accuracy drops along time with uniform sampling but remains relatively stable with GT moment input.
  • ...and 4 more figures