Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIP Multi-modal Hashing for Multimedia Retrieval

Jian Zhu, Mingkai Sheng, Zhangmin Huang, Jingfei Chang, Jinling Jiang, Jian Long, Cheng Luo, Lei Liu

TL;DR

This work proposes a novel CLIP Multi-modal Hashing method that employs the CLIP framework to extract both text and vision features and then fuses them to generate hash code and shows great improvement in the retrieval performance of multi-modal hashing methods.

Abstract

Multi-modal hashing methods are widely used in multimedia retrieval, which can fuse multi-source data to generate binary hash code. However, the individual backbone networks have limited feature expression capabilities and are not jointly pre-trained on large-scale unsupervised multi-modal data, resulting in low retrieval accuracy. To address this issue, we propose a novel CLIP Multi-modal Hashing (CLIPMH) method. Our method employs the CLIP framework to extract both text and vision features and then fuses them to generate hash code. Due to enhancement on each modal feature, our method has great improvement in the retrieval performance of multi-modal hashing methods. Compared with state-of-the-art unsupervised and supervised multi-modal hashing methods, experiments reveal that the proposed CLIPMH can significantly improve performance (a maximum increase of 8.38% in mAP).

CLIP Multi-modal Hashing for Multimedia Retrieval

TL;DR

This work proposes a novel CLIP Multi-modal Hashing method that employs the CLIP framework to extract both text and vision features and then fuses them to generate hash code and shows great improvement in the retrieval performance of multi-modal hashing methods.

Abstract

Multi-modal hashing methods are widely used in multimedia retrieval, which can fuse multi-source data to generate binary hash code. However, the individual backbone networks have limited feature expression capabilities and are not jointly pre-trained on large-scale unsupervised multi-modal data, resulting in low retrieval accuracy. To address this issue, we propose a novel CLIP Multi-modal Hashing (CLIPMH) method. Our method employs the CLIP framework to extract both text and vision features and then fuses them to generate hash code. Due to enhancement on each modal feature, our method has great improvement in the retrieval performance of multi-modal hashing methods. Compared with state-of-the-art unsupervised and supervised multi-modal hashing methods, experiments reveal that the proposed CLIPMH can significantly improve performance (a maximum increase of 8.38% in mAP).

Paper Structure

This paper contains 12 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Methodology
  4. Deep Multi-modal Hashing Network
  5. Loss Funtion
  6. Experiments
  7. Evaluation Datasets and Metrics
  8. Baseline
  9. Analysis of Experimental Results
  10. Ablation Studies
  11. Convergence Analysis
  12. Conclusion and Future Work

Figures (4)

  • Figure 1: A schematic illustration of CLIP framework. A standard vision backbone network and a text backbone network extract vision (Modal-1) and text (Modal-2) features, respectively. With these two backbones, the CLIP framework learns a vision-text pair through contrastive learning.
  • Figure 2: A schematic illustration of the proposed CLIPMH method for Multi-modal retrieval. The Vision-/Text-CLIP backbones extract vision/text features, respectively, and then these two individual features are passed through the concatenation layer before feeding them into the Multi-modal Fusion module. Finally, based on the fused features, the Hash layer generates the hash codes.
  • Figure 3: The training loss and test mAP curves on MS COCO dataset.
  • Figure 4: The training loss and test mAP curves on MIR-Flickr25K dataset.