MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation

TL;DR

MedViLaM addresses the need for a single, generalizable model that can jointly process multiple medical modalities and tasks with interpretable outputs. By coupling frozen visual encoders with a large language model through a cross-modal Q-former and instruction tuning, it supports classification, localization, grounding, and radiology report generation within a unified framework. A new MultiMedBench benchmark and an instruction-tuned multi-task dataset enable robust evaluation of cross-task generalizability, including zero-shot and few-shot settings, across radiology and non-radiology domains, video, and audio modalities. Across 12+ tasks and diverse datasets, MedViLaM achieves competitive performance, demonstrates zero-shot reasoning, and gains radiologist-validated explainability, suggesting meaningful potential for real-world clinical decision support.

Abstract

Medicine is inherently multimodal and multitask, with diverse data modalities spanning text, imaging. However, most models in medical field are unimodal single tasks and lack good generalizability and explainability. In this study, we introduce MedViLaM, a unified vision-language model towards a generalist model for medical data that can flexibly encode and interpret various forms of medical data, including clinical language and imaging, all using the same set of model weights. To facilitate the creation of such multi-task model, we have curated MultiMedBench, a comprehensive pretaining dataset and benchmark consisting of several distinct tasks, i.e., continuous question-answering, multi-label disease classification, disease localization, generation and summarization of radiology reports. MedViLaM demonstrates strong performance across all MultiMedBench tasks, frequently outpacing other generalist models by a significant margin. Additionally, we present instances of zero-shot generalization to new medical concepts and tasks, effective transfer learning across different tasks, and the emergence of zero-shot medical reasoning.

Figures (7)

• Figure 1: Illustration of MedViLaM. The joint training of multi-tasks from multi-modality is presented to improve the visual grounding at various levels of granularity. During training, we only train the linear projection matrix (as indicated with dashed boxes) while keeping the parameters of Vit and Vicuna frozen.
• Figure 2: Four modalities (image, video, volume, audio) and their associated datasets and multi-tasks: classification (CLS), segmentation (SEG), visual grounding (VG), VQA. The datasets and task objects share the same color.
• Figure 3: Model performance of Multi-label Classification task on our chest X-ray benchmark. Three radiologists independently evaluated the 12 classes of disease labels. Detailed AUC and F1 score of each disease on 10 test sets from 5 public large datasets and five private datasets of Chest X-ray images are described in the supplementary materials.
• Figure 4: The performance of our model on the 3D coronary artery visual grounding task is as follows: (a) A demo of the visual grounding task dialogue; (b) Localization performance of different cardiac structures and plaques on different coronary artery branches across various datasets, using ACC and mIoU as evaluation metrics.
• Figure 5: Performance of our model on classification and visual grounding tasks (using box or points) for endoscopic videos. The model is capable of classifying and localizing specific foreign objects and surgical instruments appearing in the videos.