Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding

TL;DR

This work introduces a visual frame-level gate mechanism that incorporates holistic textual information and uses cross-modal alignment loss to learn the fine-grained correlation between query and relevant frames and regularizes the effect of individual word tokens.

Abstract

Video Temporal Grounding (VTG) aims to identify visual frames in a video clip that match text queries. Recent studies in VTG employ cross-attention to correlate visual frames and text queries as individual token sequences. However, these approaches overlook a crucial aspect of the problem: a holistic understanding of the query sentence. A model may capture correlations between individual word tokens and arbitrary visual frames while possibly missing out on the global meaning. To address this, we introduce two primary contributions: (1) a visual frame-level gate mechanism that incorporates holistic textual information, (2) cross-modal alignment loss to learn the fine-grained correlation between query and relevant frames. As a result, we regularize the effect of individual word tokens and suppress irrelevant visual frames. We demonstrate that our method outperforms state-of-the-art approaches in VTG benchmarks, indicating that holistic text understanding guides the model to focus on the semantically important parts within the video.

Figures (4)

• Figure 1: This example shows the critical role of holistic text understanding in Video Temporal Grounding. Unlike previous works that do not take holistic text understanding into account, our method effectively filters out frames that do not correspond to the full context of the query. Here, our model does not predict the final frames due to the absence of the helmet and shades mentioned in the query.
• Figure 2: The pipeline of our framework consists of four components: feature extraction, cross-modal interaction, fine-grained alignment loss, and prediction. First, we extract visual and text features with frozen pre-trained encoders. Since the task requires cross-modal understanding and suppression of irrelevant information, we incorporate the gated cross-attention mechanism for the cross-modal interaction. The encoded features of cross-modal interaction are leveraged through the fine-grained alignment loss, which guides the model to enhance cross-modal alignment. Finally, the visual and textual representations from this aligned embedding space are fed into the prediction section to produce task-specific outputs.
• Figure 3: Qualitative results of predictions on QVHighlights validation split. We show the effectiveness of our method compared to the baseline, QD-DETR. From top to bottom are the text queries, along with the predicted moments and highlights corresponding to each method.
• Figure 4: Extended qualitative results on the QVHighlights validation split, showcasing our method's effectiveness in comparison to the baseline, QD-DETR. Displayed from top to bottom are the text queries, along with the corresponding predictions of moments and highlights for each method.