The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning

TL;DR

The paper identifies spurious correlations between text queries and background video frames as a key obstacle to precise moment localization. It proposes TD-DETR, a dynamic learning framework that (1) synthesizes dynamic context via a Video Synthesizer to create challenging, contextually varied target moments, and (2) enhances text–dynamics alignment through a Temporal Dynamic Tokenizer and a text-dynamics cross-attention, producing dynamic representations $T$ and $T'$. The method uses a transformer encoder–decoder with Hungarian matching and losses including $\\ ext{L}_{L1}$, $\\mathcal{L}_{gIoU}$, and $\\mathcal{L}_{cls}$, achieving state-of-the-art results on QVHighlights and Charades-STA and demonstrating reduced spurious correlations in extensive ablations. The findings suggest that dynamic contextualization and explicit text–dynamic alignment significantly improve temporal span accuracy and robustness, with strong generalization across architectures and feature representations.

Abstract

Given a textual query along with a corresponding video, the objective of moment retrieval aims to localize the moments relevant to the query within the video. While commendable results have been demonstrated by existing transformer-based approaches, predicting the accurate temporal span of the target moment is still a major challenge. This paper reveals that a crucial reason stems from the spurious correlation between the text query and the moment context. Namely, the model makes predictions by overly associating queries with background frames rather than distinguishing target moments. To address this issue, we propose a dynamic learning approach for moment retrieval, where two strategies are designed to mitigate the spurious correlation. First, we introduce a novel video synthesis approach to construct a dynamic context for the queried moment, enabling the model to attend to the target moment of the corresponding query across dynamic backgrounds. Second, to alleviate the over-association with backgrounds, we enhance representations temporally by incorporating text-dynamics interaction, which encourages the model to align text with target moments through complementary dynamic representations. With the proposed method, our model significantly alleviates the spurious correlation issue in moment retrieval and establishes new state-of-the-art performance on two popular benchmarks, \ie, QVHighlights and Charades-STA. In addition, detailed ablation studies and evaluations across different architectures demonstrate the generalization and effectiveness of the proposed strategies. Our code will be publicly available.

Figures (6)

• Figure 1: (a) Comparison of moment retrieval models under normal and spurious correlation videos by masking the content of target clips of video. We found the existing works are suffering from a crucial reason stems from the spurious correlation between the text queries and the moment context. Baselines predict the Spurious GT even if the target moments are masked. In contrast, TD-DETR predicts the segment near the mask with lower confidence. (b) To verify the issue of spurious correlation, we introduce the Spurious mAP as the metric. Our model achieves the best ratio of mAP to Spurious mAP.
• Figure 2: Overview of the proposed TD-DETR. Video pairs are sampled by their similarity and then forwarded into the Video Synthesizer to generate Dynamic Context (Section \ref{['sec::VSDC']}), Then, the Dynamic Context, i.e., synthesized pairs, are enhanced by Temporal Dynamic Tokenization and interact with textual information (Section \ref{['sec::TDT']}). Finally, the enhanced video tokens are sent to the transformer encoder-decoder with prediction heads to predict moments (Section \ref{['sec::network']}).
• Figure 3: Illustration of Video Synthesis with Dynamic Context. The numbers in the boxes represent the token indices, indicating their sequential order. Vision tokens are selected based on their selection probabilities and concatenated while preserving the completeness of the ground truth spans, i.e., selection probabilities of ground truth spans are set to 1. The synthesized videos maintain their original length with a random bias.
• Figure 4: Visualization of results comparison between the baseline models and our TD-DETR. We display the relative position of ground truth and prediction moments in colourful boxes, where the green with GT stands for ground truth moment, the orange box with QD-DETR stands for our baseline predictions and the purple box with ours stands for our TD-DETR. To quantify the quality of predictions, we mark the start and end timestamps in the box and mark the IoU with the same colour as the models. Best read in colour.
• Figure S1: Model prediction on query "A teacher is writing on a whiteboard.". Baselines tend to predict the teacher writing on a screen instead of the target moment which indicates baseline models fail to distinguish between "screen" and "whiteboard".