Event-aware Video Corpus Moment Retrieval

TL;DR

This work proposes EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval, and introduces anchor multi-head self-attenion to encourage Transformer to capture the relevance of adjacent content in the video.

Abstract

Video Corpus Moment Retrieval (VCMR) is a practical video retrieval task focused on identifying a specific moment within a vast corpus of untrimmed videos using the natural language query. Existing methods for VCMR typically rely on frame-aware video retrieval, calculating similarities between the query and video frames to rank videos based on maximum frame similarity.However, this approach overlooks the semantic structure embedded within the information between frames, namely, the event, a crucial element for human comprehension of videos. Motivated by this, we propose EventFormer, a model that explicitly utilizes events within videos as fundamental units for video retrieval. The model extracts event representations through event reasoning and hierarchical event encoding. The event reasoning module groups consecutive and visually similar frame representations into events, while the hierarchical event encoding encodes information at both the frame and event levels. We also introduce anchor multi-head self-attenion to encourage Transformer to capture the relevance of adjacent content in the video. The training of EventFormer is conducted by two-branch contrastive learning and dual optimization for two sub-tasks of VCMR. Extensive experiments on TVR, ANetCaps, and DiDeMo benchmarks show the effectiveness and efficiency of EventFormer in VCMR, achieving new state-of-the-art results. Additionally, the effectiveness of EventFormer is also validated on partially relevant video retrieval task.

Figures (6)

• Figure 1: In VCMR, the relevant part corresponding to the query is the moment. While the frame-aware method utilizes frames for retrieval, our event-aware approach adopts events as the retrieval unit, ensuring a more comprehensive capture of moment information.
• Figure 2: Video retriever: the hierarchical encoding of events involves interactions at both the frame and event levels, where the events are extracted by event reasoning module and Transformer for frames or events is augmented with anchor attention.
• Figure 3: Two-branch sampling for video retriever.
• Figure 4: Dual optimization for moment localizer.
• Figure 6: Moment Localizer: both frame outputs and event outputs undergo optimization during training, while only frame outputs are utilized for prediction.