Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A review of deep learning-based information fusion techniques for multimodal medical image classification

Yihao Li, Mostafa El Habib Daho, Pierre-Henri Conze, Rachid Zeghlache, Hugo Le Boité, Ramin Tadayoni, Béatrice Cochener, Mathieu Lamard, Gwenolé Quellec

TL;DR

This article surveys deep learning-based multimodal fusion for medical image classification, clarifying how input, intermediate, and output fusion strategies can be used to combine information from multiple imaging modalities. It introduces a five-strategy taxonomy (including a detailed breakdown of intermediate fusion into single-level, hierarchical, and attention-based variants) and reviews representative architectures, datasets, and empirical findings. The review highlights emerging trends such as Transformer-based fusion, cross-modal attention, and image-text pretraining, while addressing challenges like incomplete data and the need for neural architecture search. Its synthesis aids researchers in selecting appropriate fusion paradigms, datasets, and training strategies to advance robust, scalable multimodal medical AI. The practical impact lies in guiding future development toward more accurate, interpretable, and clinically applicable diagnostic tools across diverse medical domains.

Abstract

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

A review of deep learning-based information fusion techniques for multimodal medical image classification

TL;DR

This article surveys deep learning-based multimodal fusion for medical image classification, clarifying how input, intermediate, and output fusion strategies can be used to combine information from multiple imaging modalities. It introduces a five-strategy taxonomy (including a detailed breakdown of intermediate fusion into single-level, hierarchical, and attention-based variants) and reviews representative architectures, datasets, and empirical findings. The review highlights emerging trends such as Transformer-based fusion, cross-modal attention, and image-text pretraining, while addressing challenges like incomplete data and the need for neural architecture search. Its synthesis aids researchers in selecting appropriate fusion paradigms, datasets, and training strategies to advance robust, scalable multimodal medical AI. The practical impact lies in guiding future development toward more accurate, interpretable, and clinically applicable diagnostic tools across diverse medical domains.

Abstract

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.
Paper Structure (31 sections, 23 figures, 5 tables)

This paper contains 31 sections, 23 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Context
  3. Traditional methods
  4. Development trends
  5. Paper selection
  6. Highlights
  7. Multimodal medical images
  8. Imaging modalities
  9. Multimodal image datasets
  10. Multimodal classification pipeline
  11. Pre-processing
  12. Information fusion
  13. Deep learning backbone
  14. Final classifier
  15. Evaluation metrics
  16. ...and 16 more sections

Figures (23)

  • Figure 1: Overview and proportion of deep learning-based information fusion techniques for multimodal medical image classification presented in this paper.
  • Figure 2: Number of publications on medical multimodal image classification. Per-year statistics obtained using PubMed from 2016 to 2023.
  • Figure 3: Number of publications dealing with medical multimodal image classification on human organs, from 2016 to 2023. Tags: organ, number of publications, percentage.
  • Figure 4: (a) Unimodal classification task flow and different types of multimodal fusion based on the level in which they perform information fusion. (b) Information fusion networks for the three types of multimodal fusion, inputs to information fusion, and the implementation of information fusion.
  • Figure 5: (a)-(c) are the images of PET, CT, and MRI. (d)-(g) are the different sequences of MRI. Images from zhou2019review.
  • ...and 18 more figures