LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detection

TL;DR

LD-DETR tackles three core problems in Video Moment Retrieval and Highlight Detection: overlapping semantic information across samples, limited local video feature extraction, and suboptimal decoding of multimodal fusion. It introduces Distill Align to curb semantic overlap during alignment, Convolutional Fuser to efficiently extract local multimodal features, and Loop Decoder to iteratively refine decoding without overfitting. The approach achieves state-of-the-art results on QVHighlights, Charades-STA, and TACoS, with ablations validating the effectiveness of each component. This plug-and-play framework offers a scalable path toward more accurate and robust multimodal moment localization in video-plus-text tasks.

Abstract

Video Moment Retrieval and Highlight Detection aim to find corresponding content in the video based on a text query. Existing models usually first use contrastive learning methods to align video and text features, then fuse and extract multimodal information, and finally use a Transformer Decoder to decode multimodal information. However, existing methods face several issues: (1) Overlapping semantic information between different samples in the dataset hinders the model's multimodal aligning performance; (2) Existing models are not able to efficiently extract local features of the video; (3) The Transformer Decoder used by the existing model cannot adequately decode multimodal features. To address the above issues, we proposed the LD-DETR model for Video Moment Retrieval and Highlight Detection tasks. Specifically, we first distilled the similarity matrix into the identity matrix to mitigate the impact of overlapping semantic information. Then, we designed a method that enables convolutional layers to extract multimodal local features more efficiently. Finally, we fed the output of the Transformer Decoder back into itself to adequately decode multimodal information. We evaluated LD-DETR on four public benchmarks and conducted extensive experiments to demonstrate the superiority and effectiveness of our approach. Our model outperforms the State-Of-The-Art models on QVHighlight, Charades-STA and TACoS datasets. Our code is available at https://github.com/qingchen239/ld-detr.

Figures (10)

• Figure 1: Loop Decoder makes Video Moment Retrieval more accurate. We visualize the Video Moment Retrieval results corresponding to the output of Loop Decoder at each loop. As the number of loops increases, the prediction gets closer and closer to the ground truth. This experiment uses Moment-DETR lei2021detecting as the baseline.
• Figure 2: The overall framework of several recent proposed models. Those methods mainly improve the model from three perspectives: Aligning Multimodal Features, Fusing and Extracting Multimodal Features, and Decoding Multimodal Information.
• Figure 3: The overall framework of our model LD-DETR. For the methods marked with an ice cube, their parameters are not affected by gradient descent during training.
• Figure 4: The structure of the Distill Align.
• Figure 5: The structure of the Convolutional Fuser.