Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EgoIntent: An Egocentric Step-level Benchmark for Understanding What, Why, and Next

Ye Pan, Chi Kit Wong, Yuanhuiyi Lyu, Hanqian Li, Jiahao Huo, Jiacheng Chen, Lutao Jiang, Xu Zheng, Xuming Hu

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable video reasoning capabilities across diverse tasks. However, their ability to understand human intent at a fine-grained level in egocentric videos remains largely unexplored. Existing benchmarks focus primarily on episode-level intent reasoning, overlooking the finer granularity of step-level intent understanding. Yet applications such as intelligent assistants, robotic imitation learning, and augmented reality guidance require understanding not only what a person is doing at each step, but also why and what comes next, in order to provide timely and context-aware support. To this end, we introduce EgoIntent, a step-level intent understanding benchmark for egocentric videos. It comprises 3,014 steps spanning 15 diverse indoor and outdoor daily-life scenarios, and evaluates models on three complementary dimensions: local intent (What), global intent (Why), and next-step plan (Next). Crucially, each clip is truncated immediately before the key outcome of the queried step (e.g., contact or grasp) occurs and contains no frames from subsequent steps, preventing future-frame leakage and enabling a clean evaluation of anticipatory step understanding and next-step planning. We evaluate 15 MLLMs, including both state-of-the-art closed-source and open-source models. Even the best-performing model achieves an average score of only 33.31 across the three intent dimensions, underscoring that step-level intent understanding in egocentric videos remains a highly challenging problem that calls for further investigation.

EgoIntent: An Egocentric Step-level Benchmark for Understanding What, Why, and Next

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable video reasoning capabilities across diverse tasks. However, their ability to understand human intent at a fine-grained level in egocentric videos remains largely unexplored. Existing benchmarks focus primarily on episode-level intent reasoning, overlooking the finer granularity of step-level intent understanding. Yet applications such as intelligent assistants, robotic imitation learning, and augmented reality guidance require understanding not only what a person is doing at each step, but also why and what comes next, in order to provide timely and context-aware support. To this end, we introduce EgoIntent, a step-level intent understanding benchmark for egocentric videos. It comprises 3,014 steps spanning 15 diverse indoor and outdoor daily-life scenarios, and evaluates models on three complementary dimensions: local intent (What), global intent (Why), and next-step plan (Next). Crucially, each clip is truncated immediately before the key outcome of the queried step (e.g., contact or grasp) occurs and contains no frames from subsequent steps, preventing future-frame leakage and enabling a clean evaluation of anticipatory step understanding and next-step planning. We evaluate 15 MLLMs, including both state-of-the-art closed-source and open-source models. Even the best-performing model achieves an average score of only 33.31 across the three intent dimensions, underscoring that step-level intent understanding in egocentric videos remains a highly challenging problem that calls for further investigation.
Paper Structure (16 sections, 7 figures, 3 tables)

This paper contains 16 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. EgoIntent
  4. Benchmark Overview
  5. Task Definition
  6. Benchmark Construction
  7. Basic Unit and Data Source.
  8. Annotation Procedure.
  9. Temporal Truncation Strategy.
  10. Annotation Quality Control.
  11. Data Statistics and Characteristics
  12. Experiments
  13. Experimental Setup
  14. Main Result
  15. Scenario-Level Analysis
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Overall performance comparison of 15 MLLMs on the EgoIntent benchmark.
  • Figure 2: Overview of the EgoIntent benchmark. Step k is shown as an example extracted from a full egocentric video. Each clip is truncated at the step end time to prevent information leakage from future frames, and annotated with three labels: local intent, global intent, and next-step plan.
  • Figure 3: Overall environment composition of our proposed EgoIntent benchmark.
  • Figure 4: Step-count distribution across procedural activities
  • Figure 5: Examples of pre-outcome truncation in EgoIntent
  • ...and 2 more figures