Improving Video Corpus Moment Retrieval with Partial Relevance Enhancement

TL;DR

The paper addresses the challenge of partial relevance in Video Corpus Moment Retrieval (VCMR), where only parts of untrimmed videos relate to a natural language query. It introduces Partial Relevance Enhanced Model (PREM), comprising a multi-modal collaborative video retriever for video retrieval and a focus-then-fuse moment localizer for moment localization, both guided by relevant-content training. Through modality-specific pooling, gates, cross-modal fusion, contrastive learning, and adversarial training, PREM achieves state-of-the-art results on TVR and DiDeMo, demonstrating the effectiveness of focusing on query-related content across modalities. The work offers a practical approach to improve cross-modal retrieval and localization in large video corpora and provides code for reproducibility.

Abstract

Video Corpus Moment Retrieval (VCMR) is a new video retrieval task aimed at retrieving a relevant moment from a large corpus of untrimmed videos using a text query. The relevance between the video and query is partial, mainly evident in two aspects:~(1)~Scope: The untrimmed video contains many frames, but not all are relevant to the query. Strong relevance is typically observed only within the relevant moment.~(2)~Modality: The relevance of the query varies with different modalities. Action descriptions align more with visual elements, while character conversations are more related to textual information.Existing methods often treat all video contents equally, leading to sub-optimal moment retrieval. We argue that effectively capturing the partial relevance between the query and video is essential for the VCMR task. To this end, we propose a Partial Relevance Enhanced Model~(PREM) to improve VCMR. VCMR involves two sub-tasks: video retrieval and moment localization. To align with their distinct objectives, we implement specialized partial relevance enhancement strategies. For video retrieval, we introduce a multi-modal collaborative video retriever, generating different query representations for the two modalities by modality-specific pooling, ensuring a more effective match. For moment localization, we propose the focus-then-fuse moment localizer, utilizing modality-specific gates to capture essential content. We also introduce relevant content-enhanced training methods for both retriever and localizer to enhance the ability of model to capture relevant content. Experimental results on TVR and DiDeMo datasets show that the proposed model outperforms the baselines, achieving a new state-of-the-art of VCMR. The code is available at \url{https://github.com/hdy007007/PREM}.

Figures (9)

• Figure 1: In VCMR task, only a small part of the untrimmed video is explicitly related to the query, i.e., the content within the target moment. And the relevance of query to information from different modalities in the video varies.
• Figure 2: The video retriever consists of two encoders, a video encoder and a query encoder. 'ME' and 'PE' represent modality and positional embedding, respectively.
• Figure 3: Relevant and negative images sampling for contrastive learning of video retriever, and positive and negative moments sampling for adversarial learning of moment localizer. The segment in the video with a red border is the query-related moment.
• Figure 4: Moment localizer contains two key components, modality-specific gates and a multi-modal fusion network to align query and multi-modal content in the video. We use the square to represent a vector.
• Figure 5: Effect of the weight of weak relevant loss on VR. SumR is sum of R@K (K = 1, 5, 10 , 100).