Augmenting Moment Retrieval: Zero-Dependency Two-Stage Learning

TL;DR

This work tackles data scarcity, boundary ambiguity, and fine-grained semantic confusion in Moment Retrieval by introducing AMR, a zero-external-dependency framework. It combines Splice-and-Boost data augmentation to create boundary-clear and semantically challenging samples with a two-stage training pipeline (cold-start on augmented data, then distillation-adaptation to real data) and a dual-path query mechanism plus discriminative contrastive loss. Empirical results on QVHighlights, Charades-STA, and TACoS show state-of-the-art or strong improvements over prior methods, validating the approach’s ability to generalize to real-world data without external data. The method reduces labeling/data requirements while boosting localization accuracy and semantic discrimination, offering a practical route to robust video-text grounding.

Abstract

Existing Moment Retrieval methods face three critical bottlenecks: (1) data scarcity forces models into shallow keyword-feature associations; (2) boundary ambiguity in transition regions between adjacent events; (3) insufficient discrimination of fine-grained semantics (e.g., distinguishing ``kicking" vs. ``throwing" a ball). In this paper, we propose a zero-external-dependency Augmented Moment Retrieval framework, AMR, designed to overcome local optima caused by insufficient data annotations and the lack of robust boundary and semantic discrimination capabilities. AMR is built upon two key insights: (1) it resolves ambiguous boundary information and semantic confusion in existing annotations without additional data (avoiding costly manual labeling), and (2) it preserves boundary and semantic discriminative capabilities enhanced by training while generalizing to real-world scenarios, significantly improving performance. Furthermore, we propose a two-stage training framework with cold-start and distillation adaptation. The cold-start stage employs curriculum learning on augmented data to build foundational boundary/semantic awareness. The distillation stage introduces dual query sets: Original Queries maintain DETR-based localization using frozen Base Queries from the cold-start model, while Active Queries dynamically adapt to real-data distributions. A cross-stage distillation loss enforces consistency between Original and Base Queries, preventing knowledge forgetting while enabling real-world generalization. Experiments on multiple benchmarks show that AMR achieves improved performance over prior state-of-the-art approaches.

Figures (4)

• Figure 1: Traditional approaches (green diamonds) trained on real data show limited performance due to data scarcity. Data augmentation models (blue diamonds) often overfit to augmented distributions, degrading real data performance. Our two-stage training strategy (red pentagram) effectively leverages augmented data while markedly improving real data performance.
• Figure 2: Overview of our Splice-and-Boost augmentation strategy. The Splice operation combines foreground segments from one video with background intervals of another to create samples with clear boundaries. Simultaneously, the Boost mechanism introduces semantically ambiguous segments into these spliced samples. This helps enhance the model's ability to distinguish between similar events.
• Figure 3: Overview of our proposed AMR framework. The training is conducted in two stages: In the first stage, we introduce a Splice-and-Boost augmentation strategy that synthesizes diverse training samples by splicing foreground segments into background videos and injecting ambiguous distractors. In the second stage, we use dual-path distillation to transfer the knowledge from the first stage, and a discriminative contrastive loss reinforces the distinction between true events and ambiguous fragments.
• Figure 4: Qualitative comparison with other methods on QVHighlights validation set.