Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval

Zequn Xie, Xin Liu, Boyun Zhang, Yuxiao Lin, Sihang Cai, Tao Jin

TL;DR

The paper tackles text-video retrieval under sparse textual queries, where relevant visual cues can be buried in background frames. It proposes HVD, a coarse-to-fine framework with FFSM for macro-frame selection and PFCM for micro-patch compression, incorporating DPC-KNN clustering and attention to derive word-entity features, and optimizes $L_{total}=L_{T_s,V_f^*}+L_{T_w,V_p^*}$ while fusing $S_{T_s,V_f^*}$ and $S_{T_w,V_p^*}$ at inference. The two novel modules enable explicit macro- and micro-level alignment, yielding state-of-the-art results across five benchmarks and robust ablations that validate the contributions. The approach provides interpretable insights into human-like visual focus and precise cross-modal alignment, with practical impact on multimodal retrieval systems. Overall, HVD demonstrates that combining macro frame selection with micro patch compression significantly enhances text-video matching under realistic, noisy conditions.

Abstract

The success of CLIP has driven substantial progress in text-video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced attention mechanism, enabling precise entity-level matching. Extensive experiments on five benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.

HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval

TL;DR

The paper tackles text-video retrieval under sparse textual queries, where relevant visual cues can be buried in background frames. It proposes HVD, a coarse-to-fine framework with FFSM for macro-frame selection and PFCM for micro-patch compression, incorporating DPC-KNN clustering and attention to derive word-entity features, and optimizes while fusing and at inference. The two novel modules enable explicit macro- and micro-level alignment, yielding state-of-the-art results across five benchmarks and robust ablations that validate the contributions. The approach provides interpretable insights into human-like visual focus and precise cross-modal alignment, with practical impact on multimodal retrieval systems. Overall, HVD demonstrates that combining macro frame selection with micro patch compression significantly enhances text-video matching under realistic, noisy conditions.

Abstract

The success of CLIP has driven substantial progress in text-video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced attention mechanism, enabling precise entity-level matching. Extensive experiments on five benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.
Paper Structure (10 sections, 6 equations, 3 figures, 4 tables)

This paper contains 10 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Preliminaries
  4. Frame Features Selection Module
  5. Patch Features Compression Module
  6. Training and Sampling
  7. Experiments
  8. Experimental Settings
  9. Experimental Performance
  10. Conclusion

Figures (3)

  • Figure 1: Motivation. Not all visual features are relevant to the text query. Blue borders indicate the relevant frames, while yellow borders indicate the irrelevant frames.
  • Figure 2: Framework. (1) CLIP Feature Extraction Module. (2) Frame Feature Selection Module: selects keyframes that best match the text from a human global perspective based on text-frame similarity. (3) Patch Feature Compression Module: compresses visual features from a human local perspective, focusing on the most informative patch regions.
  • Figure 3: Visualization. Red lines indicate the frame features that are selected for removal.