MomentMix Augmentation with Length-Aware DETR for Temporally Robust Moment Retrieval

TL;DR

This work proposes a Length-Aware Decoder, which conditions length through a novel bipartite matching process, which surpasses state-of-the-art DETR-based methods on benchmark datasets, achieving the highest R1 and mAP on QVHighlights and the highest R1@0.7 on TACoS and Charades-STA.

Abstract

Video Moment Retrieval (MR) aims to localize moments within a video based on a given natural language query. Given the prevalent use of platforms like YouTube for information retrieval, the demand for MR techniques is significantly growing. Recent DETR-based models have made notable advances in performance but still struggle with accurately localizing short moments. Through data analysis, we identified limited feature diversity in short moments, which motivated the development of MomentMix. MomentMix generates new short-moment samples by employing two augmentation strategies: ForegroundMix and BackgroundMix, each enhancing the ability to understand the query-relevant and irrelevant frames, respectively. Additionally, our analysis of prediction bias revealed that short moments particularly struggle with accurately predicting their center positions and length of moments. To address this, we propose a Length-Aware Decoder, which conditions length through a novel bipartite matching process. Our extensive studies demonstrate the efficacy of our length-aware approach, especially in localizing short moments, leading to improved overall performance. Our method surpasses state-of-the-art DETR-based methods on benchmark datasets, achieving the highest R1 and mAP on QVHighlights and the highest R1@0.7 on TACoS and Charades-STA (such as a 9.62% gain in R1@0.7 and an 16.9% gain in mAP average for QVHighlights). The code is available at https://github.com/sjpark5800/LA-DETR.

Figures (12)

• Figure 1: (a) Performance (mAP) of previous moment retrieval (MR) DETR-based methods moon2023query_QD-DETRsun2024tr_TR-DETRxiao2024bridging_UVCOM on QVHighlightstest set by the lengths of the moment. Notice that the mAP drops significantly in capturing short-moment, where MR can be best utilized. (b) Short moment performance (mAP) comparison.
• Figure 2: Data Perspective Analysis. We analyze QVHighlightsval features in ICA hyvarinen2001ica. [Left] short moments form a tighter cluster, as indicated by a smaller 95% confidence ellipse of the covariance. [Right] For all Independent Components, the Median Absolute Deviation is greater for non-short moments, as shown by the majority of points lying above $y=x$. Overall, these highlight short moments have the limited visual information.
• Figure 3: Model Perspective Analysis. We analyze the top-1 predictions of QD-DETR on the QVHighlightsval set. [Left] Percentage of predictions where the predicted center falls within the ground truth. Only 37% of center predictions for short moments have their centers within the ground truth, indicating substantial errors in center prediction. [Right] Confusion matrix displaying predicted moment lengths (x-axis) versus ground-truth moment lengths (y-axis) across various durations. Short moments (highlighted in red) show a high rate of length prediction errors.
• Figure 4: MomentMix. A two-stage mix-based data augmentation technique for generating short-moment samples. The first stage, ForegroundMix, splits a long moment into shorter clips and shuffles them. The second stage, BackgroundMix, preserves the foreground while replacing the background with randomly cropped temporal segments from other videos.
• Figure 5: Overview of the Length-Aware Decoder (LAD). The model applies a length-wise bipartite matching strategy, where predictions and ground truths of the same class are exclusively paired. This facilitates the creation of class-specific decoder queries, enhancing sequence modeling performance.