Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

TL;DR

This paper surveys medical multimodal foundation models (MMFMs), detailing how large-scale, multimodal pretraining enables integrated analysis across imaging, text, and clinical data. It analyzes datasets, model architectures, and clinical applications, highlighting proxy-task strategies (segmentation, generative, contrastive, and hybrid) for MMVFMs and the role of vision-language models like CLIP in medicine, with emphasis on local feature representation and medical caption complexity. Key contributions include classifications of dataset types (plain text, medical images, image-text pairs), discussions of radiology report generation/comprehension, diagnosis, and treatment applications, and a roadmap addressing data, computation, reliability, interpretability, regulation, and privacy. The review underscores MMFMs’ potential to improve diagnostic accuracy, tailor treatment planning, and streamline clinical workflows, while pointing to data sharing, standardization, and ethical/regulatory considerations as critical factors for real-world deployment.

Abstract

Recent advancements in deep learning have significantly revolutionized the field of clinical diagnosis and treatment, offering novel approaches to improve diagnostic precision and treatment efficacy across diverse clinical domains, thus driving the pursuit of precision medicine. The growing availability of multi-organ and multimodal datasets has accelerated the development of large-scale Medical Multimodal Foundation Models (MMFMs). These models, known for their strong generalization capabilities and rich representational power, are increasingly being adapted to address a wide range of clinical tasks, from early diagnosis to personalized treatment strategies. This review offers a comprehensive analysis of recent developments in MMFMs, focusing on three key aspects: datasets, model architectures, and clinical applications. We also explore the challenges and opportunities in optimizing multimodal representations and discuss how these advancements are shaping the future of healthcare by enabling improved patient outcomes and more efficient clinical workflows.

Figures (8)

• Figure 1: Overview of the MMFMs workflow, illustrating how diverse multimodal medical imaging data from various organs are used for training and fine-tuning MMFMs. The two main model categories, MMVFMs and MMVLFMs, are trained and then fine-tuned for downstream tasks such as segmentation, classification, detection, registration, and clinical report generation.
• Figure 2: Result of analysis of literature with the keywords 'deep learning', 'multi modality' and following four types of tasks which are most widely studied: 'Segment', 'Classification', 'Diagnosis' and 'Prediction' on PubMed search engine, which is queried on August 9, 2024.
• Figure 3: Two different architectures to fuse images under various modalities. A. Architecture in which images are fused during the input process and transferred into the fused feature space; B. Architecture in which features are first extracted into different latent spaces and then fused together.
• Figure 4: A comprehensive overview of the four primary proxy task categories within MMVFMs. The categories include Segmentation Proxy Tasks, Generative Proxy Tasks, Contrastive Proxy Tasks, and Hybrid Proxy Tasks.
• Figure 5: Overall architecture of CLIP.