Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Background-aware Moment Detection for Video Moment Retrieval

Minjoon Jung, Youwon Jang, Seongho Choi, Joochan Kim, Jin-Hwa Kim, Byoung-Tak Zhang

TL;DR

A background-aware moment detection transformer (BM-DETR) that learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries, improving moment sensitivity and enhancing overall alignments in videos.

Abstract

Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query. This task is prone to suffer the weak alignment problem innate in video datasets. Due to the ambiguity, a query does not fully cover the relevant details of the corresponding moment, or the moment may contain misaligned and irrelevant frames, potentially limiting further performance gains. To tackle this problem, we propose a background-aware moment detection transformer (BM-DETR). Our model adopts a contrastive approach, carefully utilizing the negative queries matched to other moments in the video. Specifically, our model learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries. This leads to effective use of the surrounding background, improving moment sensitivity and enhancing overall alignments in videos. Extensive experiments on four benchmarks demonstrate the effectiveness of our approach. Our code is available at: \url{https://github.com/minjoong507/BM-DETR}

Background-aware Moment Detection for Video Moment Retrieval

TL;DR

A background-aware moment detection transformer (BM-DETR) that learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries, improving moment sensitivity and enhancing overall alignments in videos.

Abstract

Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query. This task is prone to suffer the weak alignment problem innate in video datasets. Due to the ambiguity, a query does not fully cover the relevant details of the corresponding moment, or the moment may contain misaligned and irrelevant frames, potentially limiting further performance gains. To tackle this problem, we propose a background-aware moment detection transformer (BM-DETR). Our model adopts a contrastive approach, carefully utilizing the negative queries matched to other moments in the video. Specifically, our model learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries. This leads to effective use of the surrounding background, improving moment sensitivity and enhancing overall alignments in videos. Extensive experiments on four benchmarks demonstrate the effectiveness of our approach. Our code is available at: \url{https://github.com/minjoong507/BM-DETR}
Paper Structure (19 sections, 21 equations, 4 figures, 9 tables)

This paper contains 19 sections, 21 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Video Moment Retrieval
  5. Background-aware Moment Detection
  6. Architecture
  7. Encoder.
  8. Implementing the Background-aware Moment Detection.
  9. Fine-Grained Semantic Alignment.
  10. Decoder.
  11. Temporal Shifting
  12. Learning Objectives
  13. Experiments
  14. Experiment Setup
  15. Comparison with the State-of-the-Art Methods
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Top: An example of the weak alignment problem. Bottom: Comparison between traditional (left) and proposed (right) methods.
  • Figure 2: An overview of our BM-DETR framework. First, our encoder extracts multimodal features from given inputs. Then, we obtain frame attention scores for updating multimodal features. Finally, to complete VMR, our decoder predicts the target moment, and we calculate the losses from the prediction and ground-truth moment.
  • Figure 3: Left: Bold circles indicate when our sampling strategy (ST) is not applied. Right: Bold triangles indicate when temporal shifting (TS) is not applied.
  • Figure 4: Visualization of model's predictions. We present the attention score $\mathbf{o}$ (in Equation \ref{['eq: att']}) and predicted moments (red box).