An Explainable Biomedical Foundation Model via Large-Scale Concept-Enhanced Vision-Language Pre-training

TL;DR

Healthcare AI requires both diagnostic accuracy and interpretability in medical imaging. The authors introduce ConceptCLIP, a concept-enhanced vision-language model pre-trained on MedConcept-23M (23 million image-text-concept triplets) with concepts drawn from UMLS, using a dual-alignment objective that couples global image-text alignment (IT-Align) with region-concept alignment (RC-Align) to produce fine-grained, human-interpretable explanations. The model is evaluated on a comprehensive benchmark spanning 52 clinical tasks across 10 imaging modalities, demonstrating state-of-the-art performance in medical image diagnosis and strong cross-modal capabilities (retrieval, VQA, report generation, WSI analysis) while providing explainability validated by clinicians. The work advances trustworthy AI in medicine by combining large-scale, knowledge-enhanced pre-training with explicit concept-level interpretability and extensive modality coverage, and sets a foundation for real-world clinical adoption.

Abstract

The clinical adoption of artificial intelligence (AI) in medical imaging requires models that are both diagnostically accurate and interpretable to clinicians. While current multimodal biomedical foundation models prioritize performance, their black-box nature hinders explaining the decision-making process in clinically meaningful concepts. Here, we present ConceptCLIP, the first explainable biomedical foundation model that achieves state-of-the-art diagnostic accuracy while delivering human-interpretable explanations across diverse imaging modalities. We curate MedConcept-23M, the largest pre-training dataset comprising 23 million image-text-concept triplets across diverse medical modalities, where clinical concepts are derived from the Unified Medical Language System. Leveraging this dataset, we develop ConceptCLIP through a novel dual-alignment approach that simultaneously learns global image-text representations and fine-grained region-concept associations for precise and interpretable medical image analysis. We curate the most extensive evaluation benchmark for multimodal biomedical foundation models, covering 52 clinical tasks spanning 10 imaging modalities. Extensive experiments demonstrate that ConceptCLIP outperforms existing state-of-the-art multimodal biomedical foundation models. Importantly, ConceptCLIP demonstrates superior diagnostic performance while providing human-understandable explanations validated by clinical experts. As the first precise and interpretable biomedical foundation model, ConceptCLIP represents a critical milestone toward the widespread clinical adoption of AI, thereby advancing trustworthy AI in medicine.

Figures (9)

• Figure 1: Overview of the study.a, Estimated distribution of images in MedConcept-23M dataset by image types. b, Distribution of concepts in each medical image modality studied in this work. c, The wordcloud of concepts in the MedConcept-23M dataset. Word size is proportional to its occurrence in MedConcept-23M. Common nouns and verbs are ignored. Wordclouds for each modality are shown in Extended Data Fig.\ref{['fig:wordcloud_by_modality']}. d, The pre-training process of ConceptCLIP with MedConcept-23M via an image-text alignment (IT-Align) and region-concept alignment (RC-Align) learning. "CUI" refers to "Concept Unique Identifier" in UMLS. "I-T" denotes "image and text". e, Graphical illustration of the IT-Align in the pre-training process. Clinically related region-concept pairs are gradually learned to align together. f, Statistical information of the evaluation data across different datasets and modalities. g, Comparative performance of ConceptCLIP and other multi-modal biomedical foundation models on various tasks. h, ConceptCLIP can process diverse modalities and perform versatile tasks in medical image analysis and explainable AI, showcasing its outstanding capability in trustworthy AI.
• Figure 2: Medical image diagnosis results.a, The graphical illustration depicts the process of zero-shot diagnosis, diagnosis via linear probing, and diagnosis via fully fine-tuning. For zero-shot diagnosis, $\mathbf{t}_{n, \text{cls}}$ and $\mathbf{t}_{n, \text{loc}}$ denote the global text-level and local concept-level representations of class $n$, respectively, while $\mathbf{i}_{m, \text{cls}}$ and $\mathbf{i}_{m, \text{loc}}$ represent the global and local region-level representations of the $m$-th input image (as described in the Method section). b, Zero-shot performance is evaluated using 36 datasets across 10 modalities. Average AUC scores (%) are presented for each specific image modality in the plots. Significance levels at which ConceptCLIP outperforms the best competing method are indicated using a two-sided paired t-test: ***$P<0.001$; **$P < 0.01$; *$P<0.5$. c, Linear probing performance is evaluated using 30 datasets across 10 modalities, with the mean ($\pm$s.d.) reported. Experiments considered 1%, 10%, and 100% proportions of the corresponding training set. d, Fully fine-tuning performance is evaluated using 36 datasets across 10 modalities, with the mean ($\pm$s.d.) reported.
• Figure 3: Results of other medical image analysis tasks.a, This graphical illustration outlines the processes involved in five distinct medical image analysis tasks: text-to-image retrieval, image-to-text retrieval, visual question answering, pathology whole-slide image analysis, and medical report generation. b, Performance in cross-modal retrieval is assessed on two datasets, with Image-to-Text Recall (%) and Text-to-Image Recall (%) reported for each dataset. c, visual question answering tasks are evaluated using accuracy (%) across two datasets, with the mean ($\pm$s.d.) reported. d, Performance in medical report generation is presented for two datasets using metrics such as BLEU-1,2,3,4, ROUGE-L, METEOR, Micro Precision, Micro Recall, and Micro F1. e, Pathology whole-slide image analysis results are provided across nine tasks within three categories. AUC scores (%) are reported for cancer diagnosis and mutation prediction, while C-index (%) is reported for survival prediction, with the mean ($\pm$s.d.) reported.
• Figure 4: Explainability experiment results.a, The illustration of zero-shot medical concept annotation. b, The illustration of an inherently interpretable model built upon ConceptCLIP, where a linear concept layer is adopted before the final prediction. c, The comparison results of zero-shot concept annotation on datasets of four modalities, which have fine-grained concept labels, with 95% CI reported. d, The effectiveness of the local alignment strategy, where yellow and gray bars represent the medical concept annotation results with local alignment and without local alignment during inference, respectively. e, Performance comparison of inherently interpretable models built upon medical vision-language models on disease diagnosis tasks, with 95% CI reported. In addition, the orange horizontal line indicated in each sub-figure represents the result of fully fine-tuned black-box models. f, Visualization of zero-shot medical concept annotation. For each example, the focused regions are highlighted based on the gradient of image-concept similarities, where the predicted probabilities of concepts are also presented. g, Examples of concept-class association offered by the inherently interpretable model built upon ConceptCLIP. h, Results of discovery for correlations between concepts and diseases on datasets of various modalities. According to the clinical fact, concepts located above the purple horizontal line have stronger associations with the category indicated on the upper right of each figure (e.g., Tuberculosism and Malignant), while concepts below the purple line show greater relevance to the left category (e.g., Normal and Benign). We calculate the concept presence difference to obtain which concepts are more likely to appear in which image set using the zero-shot concept annotation capability of our model. i, Disease-level concept inspection. For a specific disease, disease-level concept inspection conducts global concept-disease association analysis for images from multiple datasets to get more accurate and consistent concept-based explanations.
• Figure 7: Concepts from the MedConcept-23M dataset. Wordclouds of concepts in captions to qualitatively visualize the concepts in each medical image modality. Word size is proportional to its occurrence in MedConcept-23M. Common nouns and verbs are ignored.