Multimodal Large Language Models for Medicine: A Comprehensive Survey

TL;DR

Multimodal Large Language Models for Medicine surveys the emergence of MLLMs in healthcare, detailing data modalities, architectures, and applications across medical reporting, communication, and surgery. It analyzes core pretraining and fine-tuning strategies, alignment mechanisms, and benchmarks, while addressing data scarcity and privacy concerns. The authors identify key challenges—hallucination, professionalism, and bias—and propose practical mitigations including higher-quality data, grounding, evaluation-driven feedback, edge deployment, and privacy-preserving techniques. The paper emphasizes the potential impact of domain-specific MLLMs on clinical workflows, contingent on standardized evaluation, regulatory safeguards, and scalable deployment strategies.

Abstract

MLLMs have recently become a focal point in the field of artificial intelligence research. Building on the strong capabilities of LLMs, MLLMs are adept at addressing complex multi-modal tasks. With the release of GPT-4, MLLMs have gained substantial attention from different domains. Researchers have begun to explore the potential of MLLMs in the medical and healthcare domain. In this paper, we first introduce the background and fundamental concepts related to LLMs and MLLMs, while emphasizing the working principles of MLLMs. Subsequently, we summarize three main directions of application within healthcare: medical reporting, medical diagnosis, and medical treatment. Our findings are based on a comprehensive review of 330 recent papers in this area. We illustrate the remarkable capabilities of MLLMs in these domains by providing specific examples. For data, we present six mainstream modes of data along with their corresponding evaluation benchmarks. At the end of the survey, we discuss the challenges faced by MLLMs in the medical and healthcare domain and propose feasible methods to mitigate or overcome these issues.

Figures (9)

• Figure 1: The Evolutionary Pathway of Multimodal Large Language Models with Medical Applications Highlighted.
• Figure 2: Training Workflow for Medical Multimodal Large Language Models (MLLMs) in Application Tasks.
• Figure 3: Comprehensive Diagnosis Based on Various multimodal Data.
• Figure 4: Hallucination due to object misidentification.
• Figure 5: Hallucination due to misperceived object relationships.