QD-VMR: Query Debiasing with Contextual Understanding Enhancement for Video Moment Retrieval

TL;DR

QD-VMR tackles query semantics ambiguity in Video Moment Retrieval by integrating a Global Partial Aligner for cross-modal alignment, a Query Debiasing Module to expand and contextualize queries, and a Visual Enhancement to filter video features, all feeding a DETR-based moment predictor. The approach combines partial relevance and contrastive learning losses to align video and text while expanding the query and inferring masked words to reduce bias. Empirical results on QVHighlights, Charades-STA, and TACoS demonstrate state-of-the-art performance, with detailed ablations confirming the individual contributions of GPA, QDM, and VE. The work advances retrieval accuracy under semantic ambiguity and offers a practical path toward robust VMR, with plans to release code for wider adoption.

Abstract

Video Moment Retrieval (VMR) aims to retrieve relevant moments of an untrimmed video corresponding to the query. While cross-modal interaction approaches have shown progress in filtering out query-irrelevant information in videos, they assume the precise alignment between the query semantics and the corresponding video moments, potentially overlooking the misunderstanding of the natural language semantics. To address this challenge, we propose a novel model called \textit{QD-VMR}, a query debiasing model with enhanced contextual understanding. Firstly, we leverage a Global Partial Aligner module via video clip and query features alignment and video-query contrastive learning to enhance the cross-modal understanding capabilities of the model. Subsequently, we employ a Query Debiasing Module to obtain debiased query features efficiently, and a Visual Enhancement module to refine the video features related to the query. Finally, we adopt the DETR structure to predict the possible target video moments. Through extensive evaluations of three benchmark datasets, QD-VMR achieves state-of-the-art performance, proving its potential to improve the accuracy of VMR. Further analytical experiments demonstrate the effectiveness of our proposed module. Our code will be released to facilitate future research.

Figures (4)

• Figure 1: We analyze the query bias issues during VMR. (a) We use the global partial aligner to align the target moments in the video with the parts of the query that need to be focused. (b) We use the query debiasing module to generate the debiased query, with the underlined words being enhanced and the [bracketed] parts being expansions.
• Figure 2: An overview of QD-VMR. First, the video and query are processed through pre-trained feature extractors to obtain the corresponding features. After being mapped to the same space, a Global Partial Aligner is adopted to align the video and query features. Then, a Query Debiasing Module and a Visual Enhancement are employed to de-bias the query features and enhance the video features related to the query, respectively. Finally, a DETR structure is implemented for result prediction. The Query Debiasing Module consists of two parts: Query Expansion and Contextual Understanding Enhancement. Additionally, the encoder in the Query Expansion and the encoder in the DETR share weights.
• Figure 3: Visualization on Qvighlights val split. QD-VMR can have a better understanding of the key information in the query.
• Figure 4: The impact of global partial aligner loss coefficient $\lambda_{\text{GPA}}$ and contextual understanding enhancement loss coefficient $\lambda_w$ on QVHighlights val set without audio features.