Dual Learning with Dynamic Knowledge Distillation and Soft Alignment for Partially Relevant Video Retrieval

TL;DR

The paper tackles Partially Relevant Video Retrieval (PRVR), where untrimmed videos contain both relevant and irrelevant content. It proposes DL-DKD++—a dual-branch student (inheritance and exploration) learning from a CLIP teacher via dynamic knowledge distillation and soft target alignment—to better capture partial video-text relevance. Key contributions include a dynamic soft-target mechanism, distribution-based knowledge distillation, and a flexible two-branch framework that jointly leverages large-scale priors and dataset-specific signals. Experiments on TVR, ActivityNet-Captions, and Charades-STA show state-of-the-art results and strong ablations, with analyses confirming the benefits of soft targets, dynamic distillation, and branch complementarity. The approach offers a practical pathway to transfer rich vision-language priors to domain-specific PRVR tasks with robust performance across datasets.

Abstract

Almost all previous text-to-video retrieval works ideally assume that videos are pre-trimmed with short durations containing solely text-related content. However, in practice, videos are typically untrimmed in long durations with much more complicated background content. Therefore, in this paper, we focus on the more practical yet challenging task of Partially Relevant Video Retrieval (PRVR), which aims to retrieve partially relevant untrimmed videos with the given query. To tackle this task, we propose a novel framework that distills generalization knowledge from a powerful large-scale vision-language pre-trained model and transfers it to a lightweight, task-specific PRVR network. Specifically, we introduce a Dual Learning framework with Dynamic Knowledge Distillation (DL-DKD++), where a large teacher model provides supervision to a compact dual-branch student network. The student model comprises two branches: an inheritance branch that absorbs transferable knowledge from the teacher, and an exploration branch that learns task-specific information from the PRVR dataset to address domain gaps. To further enhance learning, we incorporate a dynamic soft-target construction mechanism. By replacing rigid hard-target supervision with adaptive soft targets that evolve during training, our method enables the model to better capture the fine-grained, partial relevance between videos and queries. Experiment results demonstrate that our proposed model achieves state-of-the-art performance on TVR, ActivityNet, and Charades-STA datasets for PRVR. The code is available at https://github.com/HuiGuanLab/DL-DKD.

Figures (9)

• Figure 1: (a) Compared to mainstream T2VR dataset, PRVR datasets involve much longer untrimmed videos; (b) Performance comparisons between the state-of-the-art (SOTA) dong2022prvr and the vanilla CLIP-basedmax2022guide methods, where CLIP shows huge performance divergences for different PRVR datasets; (c) Partial relevance inherently varies in degree, and soft targets better capture this characteristic than hard binary targets.
• Figure 2: An overview of our proposed DL-DKD++ framework. Given an untrimmed video and a query input, we design a teacher-student network to transfer the knowledge of the large-scale pre-training model CLIP to the PRVR task. In detail, the CLIP model serves as the teacher to provide generalized knowledge. A dual-branch student model is then devised to control the learning effect on knowledge transfer with a dynamic knowledge distillation strategy. In particular, an inheritance student branch is introduced to absorb the beneficial knowledge of the teacher model, while an exploration student branch is utilized to explore the task-specific property of the training data. Additionally, a dynamic soft targets construction module is introduced to convert the hard targets used in the conventional InfoNCE loss into soft targets. This allows the model to learn more reasonable partial relevance across different video-text samples.
• Figure 3: Different decay strategies during the training.
• Figure 4: Performance of different models on various types of queries. Queries are grouped according to their M/V. The smaller M/V indicates more challenging of queries.
• Figure 5: Complementarity between the two branches across three datasets. A lower correlation coefficient indicates that the two branches are more complementary.