QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries

TL;DR

This work addresses the lack of annotated data for NL-driven video moment retrieval and highlight detection by introducing QVHighlights, a large, diverse dataset with NL queries, multiple moments per video, and per-clip saliency scores. It proposes Moment-DETR, an end-to-end transformer that treats moment localization as set prediction and jointly learns saliency for highlights, removing hand-crafted pre/post-processing. The study demonstrates competitive performance of Moment-DETR against baselines, with substantial gains from weakly supervised ASR pretraining, and provides extensive ablations and analyses to guide future research. Overall, QVHighlights and Moment-DETR establish a new, unified benchmark and strong baseline for query-driven video understanding tasks with practical implications for video search and browsing.

Abstract

Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, MomentDETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr

Figures (11)

• Figure 1: QVHighlights examples. We show localized moments in dashed green boxes. The highlightness (or saliency) scores from 3 different annotators are shown under the frames as colored bars, with height and color intensity proportional to the scores.
• Figure 2: Distribution of moment lengths (left) and normalized (by video duration) center timestamps (right). The moments vary greatly in length, and they distribute almost evenly along the videos.
• Figure 3: Moment-DETR model overview. The architecture is simple, with a transformer encoder-decoder and three prediction heads for predicting saliency scores, fore-/back-ground scores and moment coordinates. For brevity, the video and text feature extractors are not shown in this figure.
• Figure 4: Visualization of all moment span predictions for all the 1550 videos on QVHighlights val split, for all the 10 moment query slots in Moment-DETR decoder. x-axis denotes the normalized moment span center coordinates w.r.t. the videos, y-axis denotes the normalized moment span width (also indicated by color). We observe that each slot learns to predict moments in different temporal locations and different widths. For example, the first slot mostly predicts short moments near the beginning of the videos, while the second slot mostly predicts short moments near the end.
• Figure 5: Prediction visualization. Predictions are shown in solid red boxes or lines, ground-truth are indicated by dashed green lines. Top row shows a correct prediction, bottom row shows a failure.