Moment of Untruth: Dealing with Negative Queries in Video Moment Retrieval

TL;DR

This work addresses the fragility of video moment retrieval models when faced with negative queries by introducing Negative-Aware Video Moment Retrieval (NA-VMR), which jointly performs moment localization and negative query rejection. It formalizes In-Domain and Out-of-Domain negatives, proposes sampling strategies (ID negatives via intra-dataset shuffling and CLIP-based filtering; OOD negatives via LLMs across broad topics), and integrates a binary relevance head into transformer-based VMR models (e.g., UniVTG) to predict a relevance score $\tilde{y}$ alongside start/end times. NA-VMR achieves high negative rejection accuracy (avg. $98.4\%$) while keeping moment retrieval performance within $3.87\%$ of Recall@1, validated on QVHighlights and Charades-STA. The work demonstrates that training with both ID and OOD negatives yields the best trade-off between robustness to negatives and retrieval accuracy, outperforming SVM baselines and contributing benchmarks and methodology for trustworthy video-language systems.

Abstract

Video Moment Retrieval is a common task to evaluate the performance of visual-language models - it involves localising start and end times of moments in videos from query sentences. The current task formulation assumes that the queried moment is present in the video, resulting in false positive moment predictions when irrelevant query sentences are provided. In this paper we propose the task of Negative-Aware Video Moment Retrieval (NA-VMR), which considers both moment retrieval accuracy and negative query rejection accuracy. We make the distinction between In-Domain and Out-of-Domain negative queries and provide new evaluation benchmarks for two popular video moment retrieval datasets: QVHighlights and Charades-STA. We analyse the ability of current SOTA video moment retrieval approaches to adapt to Negative-Aware Video Moment Retrieval and propose UniVTG-NA, an adaptation of UniVTG designed to tackle NA-VMR. UniVTG-NA achieves high negative rejection accuracy (avg. $98.4\%$) scores while retaining moment retrieval scores to within $3.87\%$ Recall@1. Dataset splits and code are available at https://github.com/keflanagan/MomentofUntruth

Figures (9)

• Figure 1: Video moment retrieval models are designed to predict start and end times in a video given a query sentence. Regardless of whether the text query is positive (exists in the video), in-domain negative (from the same domain but not present) or out-of-domain negative (from an entirely different scenario), current models such as UniVTG still produce a start-end time prediction.
• Figure 2: In-Domain and Out-of-Domain negative queries alongside a corresponding positive video-sentence pair.
• Figure 3: The classification head for NA-VMR is added to a Video Moment Retrieval model (UniVTG in this case) via summation of the indicator and saliency scores, which are then passed through a recurrent layer and a feed forward layer before producing a single value output for classification.
• Figure 4: Histograms of indicator and saliency scores from UniVTG, CG-DETR and QD-DETR on the positive and negative queries. Bottom row: Indicator and saliency scores from UniVTG-NA.
• Figure 5: Qualitative result of UniVTG-NA on QVHighlights.