Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, Heng Tao Shen

TL;DR

Cross-modal retrieval addresses the need to semantically align heterogeneous modalities at scale. The paper delivers a comprehensive taxonomy spanning unsupervised and supervised real-value and hashing approaches, plus extensive coverage of methods operating beyond text-image data, including text-video, text-audio, and 3D modalities. It highlights Vision-Language Pre-training and Cross-Modal Generation as pivotal recent directions, and provides benchmarks, evaluation metrics, and an open-source toolbox to accelerate research. The discussion emphasizes efficiency, robustness to uncertainty, and practical deployment considerations, offering concrete directions for future work and impactful applications.

Abstract

With the exponential surge in diverse multi-modal data, traditional uni-modal retrieval methods struggle to meet the needs of users seeking access to data across various modalities. To address this, cross-modal retrieval has emerged, enabling interaction across modalities, facilitating semantic matching, and leveraging complementarity and consistency between heterogeneous data. Although prior literature has reviewed the field of cross-modal retrieval, it suffers from numerous deficiencies in terms of timeliness, taxonomy, and comprehensiveness. This paper conducts a comprehensive review of cross-modal retrieval's evolution, spanning from shallow statistical analysis techniques to vision-language pre-training models. Commencing with a comprehensive taxonomy grounded in machine learning paradigms, mechanisms, and models, the paper delves deeply into the principles and architectures underpinning existing cross-modal retrieval methods. Furthermore, it offers an overview of widely-used benchmarks, metrics, and performances. Lastly, the paper probes the prospects and challenges that confront contemporary cross-modal retrieval, while engaging in a discourse on potential directions for further progress in the field. To facilitate the ongoing research on cross-modal retrieval, we develop a user-friendly toolbox and an open-source repository at https://cross-modal-retrieval.github.io.

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

TL;DR

Cross-modal retrieval addresses the need to semantically align heterogeneous modalities at scale. The paper delivers a comprehensive taxonomy spanning unsupervised and supervised real-value and hashing approaches, plus extensive coverage of methods operating beyond text-image data, including text-video, text-audio, and 3D modalities. It highlights Vision-Language Pre-training and Cross-Modal Generation as pivotal recent directions, and provides benchmarks, evaluation metrics, and an open-source toolbox to accelerate research. The discussion emphasizes efficiency, robustness to uncertainty, and practical deployment considerations, offering concrete directions for future work and impactful applications.

Abstract

With the exponential surge in diverse multi-modal data, traditional uni-modal retrieval methods struggle to meet the needs of users seeking access to data across various modalities. To address this, cross-modal retrieval has emerged, enabling interaction across modalities, facilitating semantic matching, and leveraging complementarity and consistency between heterogeneous data. Although prior literature has reviewed the field of cross-modal retrieval, it suffers from numerous deficiencies in terms of timeliness, taxonomy, and comprehensiveness. This paper conducts a comprehensive review of cross-modal retrieval's evolution, spanning from shallow statistical analysis techniques to vision-language pre-training models. Commencing with a comprehensive taxonomy grounded in machine learning paradigms, mechanisms, and models, the paper delves deeply into the principles and architectures underpinning existing cross-modal retrieval methods. Furthermore, it offers an overview of widely-used benchmarks, metrics, and performances. Lastly, the paper probes the prospects and challenges that confront contemporary cross-modal retrieval, while engaging in a discourse on potential directions for further progress in the field. To facilitate the ongoing research on cross-modal retrieval, we develop a user-friendly toolbox and an open-source repository at https://cross-modal-retrieval.github.io.
Paper Structure (82 sections, 19 equations, 10 figures, 10 tables)

This paper contains 82 sections, 19 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Overview
  3. Text-Image Cross-Modal Retrieval
  4. Unsupervised Real-value Retrieval
  5. Canonical correlation analysis methods
  6. Topic learning methods
  7. Self-reconstructing methods
  8. CNN-RNN methods
  9. GNN methods
  10. Transformer methods
  11. VLP model methods
  12. CMG model methods
  13. Supervised Real-value Retrieval
  14. Canonical correlation analysis methods
  15. Dictionary learning methods
  16. ...and 67 more sections

Figures (10)

  • Figure 1: Illustration of cross-modal retrieval. It involves retrieving information across different modalities, such as text, image, audio, and video, using a query from any one modality. For example, a user can enter a text query to retrieve relevant images or videos associated with that query.
  • Figure 2: Diagram of heterogeneous modality gap.
  • Figure 3: A comparison of real-value and hashing-based cross-modal retrieval. While both can manage diverse multi-modal data, they differ significantly in modeling strategies, data representation, and core objectives.
  • Figure 4: The evolutionary tree of representative unsupervised real-value retrieval methods.
  • Figure 5: Comparison of graph construction ways in different cross-modal retrieval tasks. Object-oriented image-text matching primarily builds object graphs within samples, whereas other cross-modal retrieval tasks focus on constructing neighbor graphs connecting samples.
  • ...and 5 more figures