Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval

Jiajun He, Tomoki Toda

TL;DR

Formulates multimodal moment retrieval as $f(V,A,Q)=(t^s,t^e)$, linking video $V$, audio $A$, and query $Q$ to start and end times. The proposed method 2DP-2MRC uses an AV-Encoder for coarse features, a 2D-Encoder for boundary localization, and a Pointer module for initial predictions, resulting in a 2D score map that fuses coarse and fine cues. It is evaluated on the HiREST dataset and reports significant improvements over baselines, with ablations confirming the importance of each module. The work offers a practical, scalable solution for precise moment localization in multimodal video data, with potential extension to moment segmentation and video captioning.

Abstract

Moment retrieval aims to locate the most relevant moment in an untrimmed video based on a given natural language query. Existing solutions can be roughly categorized into moment-based and clip-based methods. The former often involves heavy computations, while the latter, due to overlooking coarse-grained information, typically underperforms compared to moment-based models. Hence, this paper proposes a novel 2-Dimensional Pointer-based Machine Reading Comprehension for Moment Retrieval Choice (2DP-2MRC) model to address the issue of imprecise localization in clip-based methods while maintaining lower computational complexity than moment-based methods. Specifically, we introduce an AV-Encoder to capture coarse-grained information at moment and video levels. Additionally, a 2D pointer encoder module is introduced to further enhance boundary detection for target moment. Extensive experiments on the HiREST dataset demonstrate that 2DP-2MRC significantly outperforms existing baseline models.

2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval

TL;DR

Formulates multimodal moment retrieval as , linking video , audio , and query to start and end times. The proposed method 2DP-2MRC uses an AV-Encoder for coarse features, a 2D-Encoder for boundary localization, and a Pointer module for initial predictions, resulting in a 2D score map that fuses coarse and fine cues. It is evaluated on the HiREST dataset and reports significant improvements over baselines, with ablations confirming the importance of each module. The work offers a practical, scalable solution for precise moment localization in multimodal video data, with potential extension to moment segmentation and video captioning.

Abstract

Moment retrieval aims to locate the most relevant moment in an untrimmed video based on a given natural language query. Existing solutions can be roughly categorized into moment-based and clip-based methods. The former often involves heavy computations, while the latter, due to overlooking coarse-grained information, typically underperforms compared to moment-based models. Hence, this paper proposes a novel 2-Dimensional Pointer-based Machine Reading Comprehension for Moment Retrieval Choice (2DP-2MRC) model to address the issue of imprecise localization in clip-based methods while maintaining lower computational complexity than moment-based methods. Specifically, we introduce an AV-Encoder to capture coarse-grained information at moment and video levels. Additionally, a 2D pointer encoder module is introduced to further enhance boundary detection for target moment. Extensive experiments on the HiREST dataset demonstrate that 2DP-2MRC significantly outperforms existing baseline models.
Paper Structure (14 sections, 15 equations, 4 figures, 2 tables)

This paper contains 14 sections, 15 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Problem Formulation
  4. Embedding Module
  5. AV-Encoder Module
  6. Pointer Module
  7. 2DP-Encoder Module
  8. Score Prediction
  9. Loss Function
  10. Experiments
  11. Experiment Settings and Evaluation Metrics
  12. Datasets
  13. Results and Analysis
  14. Conclusion and Future Work

Figures (4)

  • Figure 1: An example of our multimodal moment retrieval with a query in an untrimmed video. The most relevant moment is retrieved by two 1D probability matrices and a 2D probability matrix together. Note that the length of the video and the sampling rate determine the value of the short duration $\tau$ and the precision of the retrieved moments.
  • Figure 2: Overall architecture of the proposed 2DP-2MRC model.
  • Figure 3: The process of score prediction.
  • Figure 4: A qualitative example of our 2DP-2MRC and ablation models evaluated on the HIREST dataset (with 128 video clips).