Object-Shot Enhanced Grounding Network for Egocentric Video

TL;DR

OSGNet tackles egocentric NLQ grounding by enriching video representations with fine-grained object features and by exploiting wearer head movements through a shot-based contrastive learning framework. The model integrates object-level cues via a cross-attentive object encoder, a BiMamba-based multimodal fusion in a main branch, and a shot branch that performs shot segmentation and InfoNCE-based cross-modal alignment. Through pretraining on NaQ and joint fine-tuning with a localization loss and a contrastive loss, OSGNet achieves state-of-the-art performance across Ego4D-NLQ, Ego4D-Goal-Step, and TACoS, with notable gains on background-object queries. This work demonstrates the practical impact of incorporating object granularity and movement-derived shot structure for robust egocentric video grounding, enabling more accurate and context-aware query localization in real-world scenarios.

Abstract

Egocentric video grounding is a crucial task for embodied intelligence applications, distinct from exocentric video moment localization. Existing methods primarily focus on the distributional differences between egocentric and exocentric videos but often neglect key characteristics of egocentric videos and the fine-grained information emphasized by question-type queries. To address these limitations, we propose OSGNet, an Object-Shot enhanced Grounding Network for egocentric video. Specifically, we extract object information from videos to enrich video representation, particularly for objects highlighted in the textual query but not directly captured in the video features. Additionally, we analyze the frequent shot movements inherent to egocentric videos, leveraging these features to extract the wearer's attention information, which enhances the model's ability to perform modality alignment. Experiments conducted on three datasets demonstrate that OSGNet achieves state-of-the-art performance, validating the effectiveness of our approach. Our code can be found at https://github.com/Yisen-Feng/OSGNet.

Figures (9)

• Figure 1: Illustration of exocentric and egocentric video grounding, accompanied by annotated narrations for the egocentric video. Key verbs and nouns are highlighted in red.
• Figure 2: The framework of our OSGNet, which consists of four key components: (a) Object Extraction, which captures fine-grained object features; (b) Feature Extraction, where visual and textual cues are processed; (c) Main Branch, responsible for primary grounding tasks; and (d) Shot Branch, which leverages wearer movement dynamics and shot-level contrastive learning to improve localization accuracy.
• Figure 3: Illustration of captions generated by LAVILA zhao2023learning describing camera movements.
• Figure 4: Qualitative comparison with GroundVQA on Ego4D-NLQ.
• Figure 5: Illustration of captions generated by LAVILA zhao2023learning describing camera movements.