Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Granularity and Multi-modal Feature Interaction Approach for Text Video Retrieval

Wenjun Li, Shudong Wang, Dong Zhao, Shenghui Xu, Zhaoming Pan, Zhimin Zhang

TL;DR

The paper tackles text-video retrieval by addressing uneven word relevance and underutilized audio information. It introduces two modules, MGFI for multi-granularity text-frame/word-frame interactions and CMFI for audio-text cross-modal interaction, within a CLIP-based framework. The approach yields a robust video representation and an audio-augmented cross-modal signal, optimized with a dual InfoNCE loss. Empirical results on MSR-VTT, MSVD, and DiDeMo show state-of-the-art performance and strong ablation-supported gains, highlighting the benefits of coordinated multi-granularity and cross-modal fusion for TVR.

Abstract

The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.

Multi-Granularity and Multi-modal Feature Interaction Approach for Text Video Retrieval

TL;DR

The paper tackles text-video retrieval by addressing uneven word relevance and underutilized audio information. It introduces two modules, MGFI for multi-granularity text-frame/word-frame interactions and CMFI for audio-text cross-modal interaction, within a CLIP-based framework. The approach yields a robust video representation and an audio-augmented cross-modal signal, optimized with a dual InfoNCE loss. Empirical results on MSR-VTT, MSVD, and DiDeMo show state-of-the-art performance and strong ablation-supported gains, highlighting the benefits of coordinated multi-granularity and cross-modal fusion for TVR.

Abstract

The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.
Paper Structure (12 sections, 14 equations, 1 figure, 4 tables)

This paper contains 12 sections, 14 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction and background
  2. proposed Methods
  3. Overall Structure Design
  4. Multi-Granularity Feature Interaction
  5. Cross-Modal Feature Interaction
  6. Loss Function
  7. experiments and results
  8. Datasets And Metrics
  9. Implementation Details
  10. Ablation Study
  11. Comparison With State-of-the-arts
  12. conclusion

Figures (1)

  • Figure 1: Framework of our approach which is comprised of two components: MGFI aligns video and text, and CMFI complements expression of frames information in video.