Enhanced Partially Relevant Video Retrieval through Inter- and Intra-Sample Analysis with Coherence Prediction

TL;DR

This work tackles Partially Relevant Video Retrieval by recognizing and exploiting the cross-modal duality of inter-sample correlations and intra-sample redundancy. It introduces three modules—ICE for pseudo-positive cross-modal pairs, IRM for removing redundant moments, and TCP for self-supervised temporal modeling—to build a more discriminative cross-modal space. The approach yields state-of-the-art results on TVR, ActivityNet Captions, and Charades-STA and demonstrates robustness via extensive ablations and qualitative analyses. The framework is modular and plug-and-play, offering practical gains for fine-grained moment-level video-text retrieval in real-world untrimmed videos.

Abstract

Partially Relevant Video Retrieval (PRVR) aims to retrieve the target video that is partially relevant to the text query. The primary challenge in PRVR arises from the semantic asymmetry between textual and visual modalities, as videos often contain substantial content irrelevant to the query. Existing methods coarsely align paired videos and text queries to construct the semantic space, neglecting the critical cross-modal dual nature inherent in this task: inter-sample correlation and intra-sample redundancy. To this end, we propose a novel PRVR framework to systematically exploit these two characteristics. Our framework consists of three core modules. First, the Inter Correlation Enhancement (ICE) module captures inter-sample correlation by identifying semantically similar yet unpaired text queries and video moments, combining them to form pseudo-positive pairs for more robust semantic space construction. Second, the Intra Redundancy Mining (IRM) module mitigates intra-sample redundancy by mining redundant moment features and distinguishing them from query-relevant moments, encouraging the model to learn more discriminative representations. Finally, to reinforce these modules, we introduce the Temporal Coherence Prediction (TCP) module, enhancing discrimination of fine-grained moment-level semantics by training the model to predict the original temporal order of randomly shuffled video sequences. Extensive experiments demonstrate the superiority of our method, achieving state-of-the-art results.

Figures (7)

• Figure 1: In Partially Relevant Video Retrieval (PRVR), video-text samples exhibit the inherent cross-modal dual nature: (a) Inter-sample correlation: The video contains certain moments that are semantically correlated to other unpaired text queries. (b) Intra-sample redundancy: Apart from the target moment, other redundant moments in the video are irrelevant to the paired text query.
• Figure 2: Overview of the proposed framework. We systematically leverage the cross-modal dual nature in PRVR, namely inter-sample correlation and intra-sample redundancy, complemented by temporal coherence prediction, to construct a more discriminative cross-modal semantic space. The framework comprises three key components: (a) Inter Correlation Enhancement Module: This component analyzes cross-modal correlations by identifying high-similarity pairs between video moments and unpaired text queries. These pseudo-positive pairs are incorporated during training to enrich the semantic space construction. (b) Intra Redundancy Mining Module: The module extracts redundant video moments. By learning to distinguish these redundant moments and query-relevant moments, the model develops enhanced capability to focus on query-relevant visual semantics. (c) Temporal Coherence Prediction Module: Designed to complement the other components, this module improves discrimination of fine-grained moment-level semantics through a self-supervised sequence prediction task, where the model predicts the original temporal order of shuffled video frames/moments.
• Figure 3: The ICE module employs a two-stage selection process for pseudo label assignment. It first computes the similarity between unpaired video moments and text features in a mini-batch. The pairs with mutual maximum similarity are selected as candidate pairs. Then, only those pairs with a similarity higher than the threshold are retained, ensuring high-confidence pseudo labels.
• Figure 4: The pipeline of the Temporal Coherence Prediction (TCP) module. The frame features are first divided into distinct groups. Then, a subset of frame features is randomly selected and shuffled, and the group labels of frames are predicted.
• Figure 5: Visualization comparisons of retrieval results between our method, MS-SL MSSL, and GMMFormer GMMFORMER on ActivityNet Captions. The top-5 retrieval results are shown from left to right. Ground-truth videos are marked with red boxes. Zoom in for better visibility.