GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval

TL;DR

GMMFormer tackles partially relevant video retrieval by replacing costly explicit clip scanning with implicit clip modeling using multi-scale Gaussian blocks within a Transformer architecture. A Gaussian-Mixture-Model constraint focuses frame interactions on nearby frames to encode multi-length clip information, while a novel query-diverse loss preserves semantic distinctions among text queries tied to the same video. The approach achieves state-of-the-art results on TVR, ActivityNet Captions, and Charades-STA, with substantial efficiency gains over prior methods. This combination of accuracy and efficiency enhances PRVR applicability in real-world video search scenarios.

Abstract

Given a text query, partially relevant video retrieval (PRVR) seeks to find untrimmed videos containing pertinent moments in a database. For PRVR, clip modeling is essential to capture the partial relationship between texts and videos. Current PRVR methods adopt scanning-based clip construction to achieve explicit clip modeling, which is information-redundant and requires a large storage overhead. To solve the efficiency problem of PRVR methods, this paper proposes GMMFormer, a Gaussian-Mixture-Model based Transformer which models clip representations implicitly. During frame interactions, we incorporate Gaussian-Mixture-Model constraints to focus each frame on its adjacent frames instead of the whole video. Then generated representations will contain multi-scale clip information, achieving implicit clip modeling. In addition, PRVR methods ignore semantic differences between text queries relevant to the same video, leading to a sparse embedding space. We propose a query diverse loss to distinguish these text queries, making the embedding space more intensive and contain more semantic information. Extensive experiments on three large-scale video datasets (i.e., TVR, ActivityNet Captions, and Charades-STA) demonstrate the superiority and efficiency of GMMFormer. Code is available at \url{https://github.com/huangmozhi9527/GMMFormer}.

Figures (7)

• Figure 1: Traditional text-to-video retrieval pipelines (a) generate compact video embeddings and lost clip information. Previous partially relevant video retrieval pipelines (b) adopt explicit clip modeling, which is information-redundant and requires a large storage overhead. We utilize implicit clip modeling (c) to obtain compact clip embeddings, containing multi-scale clip information.
• Figure 2: The overall framework of GMMFormer.$\otimes$ denotes the matrix multiplication.
• Figure 3: The illustration of the GMMFormer block.
• Figure 4: Ablation studies of the Gaussian block on ActivityNet Captions with different types of queries. Queries are grouped according to their moment-to-video ratios (M/V). Different Gaussian blocks are good at handling different M/V groups. And a GMMFormer variant w/o any Gaussian block will perform poorly on the corresponding group.
• Figure 5: Text-clip similarity visualizations on TVR. w/o GB means a variant of GMMFormer replacing GMMFormer blocks with vanilla Transformer blocks. Note that we smooth out similarity intervals for better observation.