MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging

TL;DR

MedImageInsight introduces an open-source, generalist medical imaging embedding model with a two-tower architecture (DaViT image encoder and UniCL language encoder) that scales across 14 imaging domains. Trained on heterogeneous image-text and image-label data, it supports base image-text and image-image search, 3D image retrieval, and near-SOTA single-image report generation with a lightweight decoder, while providing ROC curves for regulatory needs and demonstrated fairness across demographics. The model achieves SOTA or human-expert performance on multiple public datasets (including CT 3D retrieval and chest X-ray, dermatology, and OCT classification/search) and shows robust 3D retrieval performance, with open-source weights to foster transparency and collaboration. Collectively, MedImageInsight advances cross-domain medical imaging AI by delivering interpretable retrieval-based evidence, compact generation capabilities, and broad domain coverage, enabling scalable research and practical deployment.

Abstract

In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.

Figures (5)

• Figure 1: Overview of the MedImageInsight foundation model. a) Chord diagram of datasets and modalities used for training and evaluation. b) MedImageInsight report findings generation performance among methods that leverage a single image on MIMIC-CXR (except multi-image benchmarks labeled with subscript "multi"). c) Radar figure of classification performance of single model (no fine-tuning) across datasets. All metrics are mAUC except for SD-198 and OCT2018 which are displayed as accuracy. Reference refers to SOTA. d) 3D retrieval results for 3D-MIR benchmark (inherent capability, no fine-tuning). TP = Tumor Presence. TS = Tumor Stage. P@N = Precision @ N. e) 3 examples of report generation predictions for chest X-ray. f) 10 examples of broad domain medical image classification from roughly 1000 classes (inherent capability, no fine-tuning).
• Figure 2: Example image-image search results for the ISIC2019 dataset. 1 query per disease label was randomly selected from the validation dataset. Correctly matching labels highlighted green, incorrectly mis-matching labels highlighted pink. NEV = Melanocytic nevus. BCC = Basal cell carcinoma. SCC = Squamous cell carcinoma. ACK = Actinic keratosis. BEK = Benign keratosis. DFB = Dermatofibroma. MEL = Melanoma. VAS = Vascular lesion.
• Figure 3: ROC curves from independent site evaluation using KNN classification on extracted image features for binary and multi-class image classification. Left column: ROC curves for binary classification. Right column: ROC curves for multi-class classification.
• Figure 4: Overview of the MedImageInsight foundation model architecture. UniCL is used as the pre-training objective function.
• Figure 5: MIMIC-CXR metrics according to performance and model size. Multi-image MIMIC-CXR benchmarks denoted with subscript "multi".