Fine-grained Spatiotemporal Grounding on Egocentric Videos

TL;DR

This work tackles the lack of pixel-level spatiotemporal grounding benchmarks for egocentric videos by introducing EgoMask, the first such benchmark, and EgoMask-Train for large-scale training. The authors develop an automatic annotation pipeline that couples SAM2-based mask generation with GPT-4o–generated referring expressions, enabling pixel-level ground truth across short, medium, and long video durations. Through extensive experiments, they show that state-of-the-art models struggle on EgoMask but reap substantial gains when fine-tuned on EgoMask-Train, while maintaining performance on exocentric benchmarks, indicating complementary value to existing datasets. Overall, EgoMask provides essential resources and insights to advance egocentric video understanding and embodied AI.

Abstract

Spatiotemporal video grounding aims to localize target entities in videos based on textual queries. While existing research has made significant progress in exocentric videos, the egocentric setting remains relatively underexplored, despite its growing importance in applications such as augmented reality and robotics. In this work, we conduct a systematic analysis of the discrepancies between egocentric and exocentric videos, revealing key challenges such as shorter object durations, sparser trajectories, smaller object sizes, and larger positional shifts. To address these challenges, we introduce EgoMask, the first pixel-level benchmark for fine-grained spatiotemporal grounding in egocentric videos. It is constructed by our proposed automatic annotation pipeline, which annotates referring expressions and object masks across short-, medium-, and long-term videos. Additionally, we create EgoMask-Train, a large-scale training dataset to facilitate model development. Experiments demonstrate that the state-of-the-art spatiotemporal grounding models perform poorly on our benchmark EgoMask, but fine-tuning on EgoMask-Train yields significant improvements, while preserving performance on exocentric datasets. Our work thus provides essential resources and insights for advancing egocentric video understanding. Our code is available at https://github.com/LaVi-Lab/EgoMask .

Figures (12)

• Figure 1: Comparison of video grounding tasks. We propose the first pixel-level benchmark for fine-grained spatiotemporal grounding in egocentric videos.
• Figure 2: Automatic Annotation Pipeline. The inputs include video frames, the annotated bounding boxes, and the object category from the Egotracks dataset. The annotation process contains two parts: (1) Mask Generation (Bottom Left): utilizing SAM2 as annotator to generate mask throughout the input frames; (2) Referring Expression Generation (Bottom Right): prompting GPT-4o to directly generate referring expressions (blue arrow) or first generate metadata about the labeled object and then adopt pre-defined templates to generate expressions (green arrow).
• Figure 3: Visualization of one example from EgoMask-Short with sampled frames. The language query is "black container bottle on the left side of a wooden table behind computer tablet". The fine-tuned models perform better than their zero-shot counterparts.
• Figure 4: Visualization of one example from EgoMask-Long with sampled frames. The language query is "small blue cylindrical container near the floor". The small target poses a challenge to existing methods.
• Figure 5: Video Length Distribution of EgoMask-Train.