When One Moment Isn't Enough: Multi-Moment Retrieval with Cross-Moment Interactions

TL;DR

The paper tackles real-world video temporal grounding where a single query maps to multiple moments. It introduces QV-M2, a fully human-annotated multi-moment dataset, and new MMR metrics to evaluate dense moment grounding. To address MMR, it proposes FlashMMR, a multi-scale, cross-modal framework with a Post-Verification module that refines and verifies candidate moments via a semantic-consistency network and structured post-processing. Empirical results show FlashMMR outperforms prior methods on QV-M2 and QVHighlights, establishing a benchmark and baseline for more realistic video grounding tasks. The work highlights the need for multi-moment framing and provides a foundation for future improvements in temporal grounding under complex, real-world settings.

Abstract

Existing Moment retrieval (MR) methods focus on Single-Moment Retrieval (SMR). However, one query can correspond to multiple relevant moments in real-world applications. This makes the existing datasets and methods insufficient for video temporal grounding. By revisiting the gap between current MR tasks and real-world applications, we introduce a high-quality datasets called QVHighlights Multi-Moment Dataset (QV-M$^2$), along with new evaluation metrics tailored for multi-moment retrieval (MMR). QV-M$^2$ consists of 2,212 annotations covering 6,384 video segments. Building on existing efforts in MMR, we propose a framework called FlashMMR. Specifically, we propose a Multi-moment Post-verification module to refine the moment boundaries. We introduce constrained temporal adjustment and subsequently leverage a verification module to re-evaluate the candidate segments. Through this sophisticated filtering pipeline, low-confidence proposals are pruned, and robust multi-moment alignment is achieved. We retrain and evaluate 6 existing MR methods on QV-M$^2$ and QVHighlights under both SMR and MMR settings. Results show that QV-M$^2$ serves as an effective benchmark for training and evaluating MMR models, while FlashMMR provides a strong baseline. Specifically, on QV-M$^2$, it achieves improvements over prior SOTA method by 3.00% on G-mAP, 2.70% on mAP@3+tgt, and 2.56% on mR@3. The proposed benchmark and method establish a foundation for advancing research in more realistic and challenging video temporal grounding scenarios. Code is released at https://github.com/Zhuo-Cao/QV-M2.

Figures (2)

• Figure 1: Dataset statistics for QV-M$^2$. (a) and (c) show the distributions of moment lengths and temporal locations, respectively; (b) reports the top-10 most frequent words in the annotations.
• Figure 2: Overview of the FlashMMR Framework. In Feature Extraction, video and query features are extracted via frozen encoders. The textual feature, combined with encoded dummy tokens, forms the key, while the video feature serves as the query and the text feature as the value in Feature Fusion Module. This produces fused features $\{f_i\}_{i=1}^{L_v}$ (in purple), where color intensity indicates semantic relevance. During inference, the fused features are directly passed into the Multi-Scale Temporal Prediction Module to generate the final prediction $\{(s_i, e_i, c_i)\}_{i=1}^{3}$. During training, the Post-Verification Module further refines the initial prediction. Specifically, the fused features are aligned with the prediction to obtain a refined confidence score $p_i$ and a self-similarity matrix $S_i$, both of which are supervised using the ground truth moments.