Unleash the Potential of CLIP for Video Highlight Detection

TL;DR

The paper tackles video highlight detection by leveraging open-world knowledge embedded in multimodal models. It introduces HL-CLIP, a detector-free framework that fine-tunes the last layers of CLIP and employs saliency pooling to produce frame-level highlight scores, with the core scoring given by $\hat{y}_{i,j} = \mathrm{sim}(h^v_{i,j}, h^t_i)$ and trained via $L_{saliency} = \frac{1}{N \cdot K} \sum_{i=1}^{N} \sum_{j=1}^{K} (\hat{y}_{i,j} - y_{i,j})^2$. The approach demonstrates state-of-the-art performance on the QVHighlight benchmark, highlighting the effectiveness and efficiency of fine-tuning pretrained multimodal encoders for time-aware video tasks, and suggesting avenues to extend to moment retrieval with future work. This work underscores the practical impact of repurposing CLIP-like models for targeted video understanding tasks, enabling robust highlight detection with fewer trainable parameters and minimal architectural changes.

Abstract

Multimodal and large language models (LLMs) have revolutionized the utilization of open-world knowledge, unlocking novel potentials across various tasks and applications. Among these domains, the video domain has notably benefited from their capabilities. In this paper, we present Highlight-CLIP (HL-CLIP), a method designed to excel in the video highlight detection task by leveraging the pre-trained knowledge embedded in multimodal models. By simply fine-tuning the multimodal encoder in combination with our innovative saliency pooling technique, we have achieved the state-of-the-art performance in the highlight detection task, the QVHighlight Benchmark, to the best of our knowledge.

Figures (1)

• Figure 1: The overall architecture of Highlight-CLIP (HL-CLIP). The last few layers of transformer encoders are finetuned to estimate the saliency score between a frame and a query. The green and blue lines denote the predicted and the ground truth saliency score of a video frame given a query, respectively.