Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval

TL;DR

This work targets the misalignment between training objectives and AP-based evaluation in video retrieval. It introduces HAP-VR, a self-supervised hierarchical framework that optimizes AP at both video and frame levels using a novel TopK-Chamfer video similarity and a gradient-friendly QuadLinear-$AP$ surrogate, complemented by frame similarity distillation from a self-supervised teacher. Empirically, HAP-VR delivers consistent gains over strong baselines on EVVE, SVD, and FIVR datasets, with notable improvements in both retrieval ($mAP$) and detection ($m{Cmu AP}$) tasks, and ablations validate the contributions of each component. The approach offers a practical pathway to align training with AP evaluation in videos, improving top-list quality and robustness to frame-level noise for real-world multimedia retrieval tasks.

Abstract

The rapid growth of online video resources has significantly promoted the development of video retrieval methods. As a standard evaluation metric for video retrieval, Average Precision (AP) assesses the overall rankings of relevant videos at the top list, making the predicted scores a reliable reference for users. However, recent video retrieval methods utilize pair-wise losses that treat all sample pairs equally, leading to an evident gap between the training objective and evaluation metric. To effectively bridge this gap, in this work, we aim to address two primary challenges: a) The current similarity measure and AP-based loss are suboptimal for video retrieval; b) The noticeable noise from frame-to-frame matching introduces ambiguity in estimating the AP loss. In response to these challenges, we propose the Hierarchical learning framework for Average-Precision-oriented Video Retrieval (HAP-VR). For the former challenge, we develop the TopK-Chamfer Similarity and QuadLinear-AP loss to measure and optimize video-level similarities in terms of AP. For the latter challenge, we suggest constraining the frame-level similarities to achieve an accurate AP loss estimation. Experimental results present that HAP-VR outperforms existing methods on several benchmark datasets, providing a feasible solution for video retrieval tasks and thus offering potential benefits for the multi-media application.

Figures (7)

• Figure 1: Average Precision (AP) measures the average ranking of positive instances within a list, providing a comprehensive evaluation of the overall performance of the retrieval results. Pair-wise training objectives focus solely on pulling the positive instances closer while repelling the negative ones, failing to align with the AP metric. In contrast, AP-based objectives ensure this alignment by rectifying the rankings of mis-ranked positive-negative pairs in the list.
• Figure 2: Two relevant videos might not exhibit consistent relevance across all frame pairs due to the obvious redundancy and noise in the temporal dimension. Specifically, only a few consecutive frames in the candidate video are relevant to a given query frame. $\bm{T_Q}$ and $\bm{T_R}$ axes represent the timelines of the query and reference videos, respectively.
• Figure 3: The architecture of our proposed framework. The data batch is processed through a feature extractor to obtain patch-level embeddings. Afterward, we compute frame-level and video-level similarity matrices utilizing spatial and temporal correlation aggregation modules in sequence. Simultaneously, the batch is fed into a pre-trained self-supervised model to generate pseudo labels that indicate frame-level relevance. Ultimately, we apply the QuadLinear-AP to both the frame-level and video-level similarity matrices and backpropagate the loss to optimize the model's parameters.
• Figure 4: The curves of Sigmoid function in Smooth-AP and surrogate loss function for positive-negative pairs in QuadLinear-AP and their derivative functions. The colored parts in (b) and (d) represent non-zero gradient areas.
• Figure 5: Video similarity distribution of relevant and irrelevant instance pairs for HAP-VR, DnS, and TCA on the DSVD set of FIVR-200K. All similarities are rescaled to $\bm{[0, 1]}$.