Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval

Zequn Xie, Boyun Zhang, Yuxiao Lin, Tao Jin

TL;DR

Video-text retrieval is limited by redundancy in final-layer vision features and underutilization of hierarchical semantics within encoders. The authors present HVP-Net, which extracts and refines multi-level frame and patch features from intermediate Vision Transformer layers and distills patch concepts with a Multi-layer Patch Processing module. They employ multi-granularity cross-modal alignment (sentence-frame, sentence-patch, word-patch) trained with symmetric InfoNCE losses across layers, and sum scores at inference. On MSR-VTT, DiDeMo, and ActivityNet, HVP-Net achieves state-of-the-art results, demonstrating the benefit of exploiting hierarchical encoder representations for robust video-text retrieval. This approach highlights the potential of mining intermediate features in pre-trained models to enhance cross-modal alignment and retrieval performance.

Abstract

Video-text retrieval (VTR) aims to locate relevant videos using natural language queries. Current methods, often based on pre-trained models like CLIP, are hindered by video's inherent redundancy and their reliance on coarse, final-layer features, limiting matching accuracy. To address this, we introduce the HVP-Net (Hierarchical Visual Perception Network), a framework that mines richer video semantics by extracting and refining features from multiple intermediate layers of a vision encoder. Our approach progressively distills salient visual concepts from raw patch-tokens at different semantic levels, mitigating redundancy while preserving crucial details for alignment. This results in a more robust video representation, leading to new state-of-the-art performance on challenging benchmarks including MSRVTT, DiDeMo, and ActivityNet. Our work validates the effectiveness of exploiting hierarchical features for advancing video-text retrieval. Our codes are available at https://github.com/boyun-zhang/HVP-Net.

Delving Deeper: Hierarchical Visual Perception for Robust Video-Text Retrieval

TL;DR

Video-text retrieval is limited by redundancy in final-layer vision features and underutilization of hierarchical semantics within encoders. The authors present HVP-Net, which extracts and refines multi-level frame and patch features from intermediate Vision Transformer layers and distills patch concepts with a Multi-layer Patch Processing module. They employ multi-granularity cross-modal alignment (sentence-frame, sentence-patch, word-patch) trained with symmetric InfoNCE losses across layers, and sum scores at inference. On MSR-VTT, DiDeMo, and ActivityNet, HVP-Net achieves state-of-the-art results, demonstrating the benefit of exploiting hierarchical encoder representations for robust video-text retrieval. This approach highlights the potential of mining intermediate features in pre-trained models to enhance cross-modal alignment and retrieval performance.

Abstract

Video-text retrieval (VTR) aims to locate relevant videos using natural language queries. Current methods, often based on pre-trained models like CLIP, are hindered by video's inherent redundancy and their reliance on coarse, final-layer features, limiting matching accuracy. To address this, we introduce the HVP-Net (Hierarchical Visual Perception Network), a framework that mines richer video semantics by extracting and refining features from multiple intermediate layers of a vision encoder. Our approach progressively distills salient visual concepts from raw patch-tokens at different semantic levels, mitigating redundancy while preserving crucial details for alignment. This results in a more robust video representation, leading to new state-of-the-art performance on challenging benchmarks including MSRVTT, DiDeMo, and ActivityNet. Our work validates the effectiveness of exploiting hierarchical features for advancing video-text retrieval. Our codes are available at https://github.com/boyun-zhang/HVP-Net.
Paper Structure (10 sections, 2 equations, 3 figures, 3 tables)

This paper contains 10 sections, 2 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Hierarchical Feature Encoding
  4. Multi-layer Patch Processing (MPP)
  5. Multi-granularity Alignment and Optimization
  6. Experiments
  7. Experimental Setup
  8. Comparison with State-of-the-Art Methods
  9. Ablation Studies and Analysis
  10. Conclusion

Figures (3)

  • Figure 1: Conceptual illustration of our HVP-Net, inspired by the human hierarchical perception process. Our model mimics this by processing visual data through shallow, mid, and deep layers to extract concepts ranging from low-level textures to high-level events.
  • Figure 2: The architecture of our HVP-Net. We first extract hierarchical frame ($F_l$) and patch ($P_l$) features from a video encoder. Then, our Multi-layer Patch Processing module refines the patch features into $\hat{P}_l$. Finally, we perform multi-granularity alignment between text and the multi-level video features ($F_l$, $\hat{P}_l$) and aggregate the layer-wise similarity scores for retrieval.
  • Figure 3: The architecture of the Multi-layer Patch Processing (MPP) module, which iteratively refines input patch features $P_l$ to $\hat{P}_l$. In each processing block, a Patch Compression Module (PCM) first distills $M$ tokens into $K$ core concepts using saliency-guided clustering. Subsequently, these concepts are refined via a cross-attention mechanism that attends back to the original tokens.