Merlin: A Vision Language Foundation Model for 3D Computed Tomography

TL;DR

Merlin tackles radiologist workload by introducing a 3D vision-language foundation model trained with multimodal clinical data (CT scans, EHR codes, and radiology reports) to understand abdominal CTs in 3D. It processes full CT volumes on a single GPU and is evaluated across six task families totaling 752 tasks, achieving strong zero-shot and adapted-task performance and outperforming several baselines on internal and external data. The work establishes data scaling laws, conducts extensive ablations, and demonstrates practical applications in disease risk stratification and radiology report generation, while detailing limitations and directions for expansion to additional anatomies and modalities. Overall, Merlin demonstrates the feasibility and impact of data-efficient, multimodal 3D radiology foundation models for scalable clinical decision support.

Abstract

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current radiologist shortage, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs). However, current medical VLMs are generally limited to 2D images and short reports, and do not leverage electronic health record (EHR) data for supervision. We introduce Merlin - a 3D VLM that we train using paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens). We evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU.

Figures (8)

• Figure 1: Overview of Merlin training and evaluation. (a) Merlin training strategy. Diagnosis codes from the EHR are used as labels for Merlin training, with a binary cross entropy loss. Radiology reports are also used for training, with an InfoNCE loss oord2018representation. Training with diagnosis codes and radiology reports is either staged or performed in a multi-task manner. Merlin is then evaluated on non-adapted tasks that can be performed without any architectural or weight modifications. These include (b) zero-shot findings classification, (c) phenotype classification, and (d) zero-shot cross-modal retrieval. Adapting Merlin enables us to perform (e) 5-year disease prediction, (f) radiology report generation, and (g) 3D semantic segmentation. All error bars are 95% confidence intervals.
• Figure 2: Zero-shot findings classification. (a) Depicts how zero-shot classification is performed where text embeddings from disease presence prompts and disease absence prompts are compare to the image embedding. (b) We compare the performance of OpenCLIP cherti2022reproducible, BioMedCLIP zhang2023biomedclip, Merlin on an internal dataset, and Merlin without radiology report splitting. We further evaluate Merlin on an external clinical dataset and the VerSe loffler2020vertebral external fracture detection dataset. (c) Performance of BioMedCLIP, Merlin, and Merlin without report splitting on the internal dataset, as well as Merlin on the external dataset, across 30 findings assessed on abdominal CT scans. (d) Merlin zero-shot classification performance improves with increasing pretraining dataset size. (e) An ablation study across various aspects of Merlin's pretraining strategy. "Rpt." is shorthand for "report" and indicated training with radiology reports only. Staged (Stg.) refers to performing weakly supervised training with EHR in a first training stage and then training with radiology reports in a second stage. This is in contrast to multi-task learning (MTL) where EHR and radiology reports are used for training simultaneously.
• Figure 3: Phenotype classification. (a) Average AUROC performance for the top 20 phenotype groups listed in order of prevalence (black line). (b) Data scaling law experiments that measure how average AUROC (top) and average AUPRC (bottom) across the 692 phenotypes scale as the amount of pretraining data varies. (c) Average AUROC (left chart) and AUPRC (right chart) across all 692 phenotypes, the top quartile of 173 phenotypes, and the botton quartile of 173 phenotypes across several baseline models. All baseline models are trained using the phenotypes in the pretraining dataset. The dashed lines denote random chance performance. Note that Merlin, which uses the best performing backbone of ResNet152, is further trained using radiology reports. (d) Average AUROC as a function of model stem hyper-parameters. We find that a smaller receptive field yields better performance. (e) Counterfactual analyses of pleural effusion classification (left; image from TCIA clark2013cancer) and splenomegaly classification (image from our internal test set). We annotate the zoomed in images by outlining the pathologies. The red lines border pathologies in the original images. The blue lines border pathologies in the counterfactual images. Counterfactual outlines are drawn over the original images with dotted lines and the original image outlines are also drawn over the counterfactual images with dotted lines. This allows comparing the size and shape of the pathologies between the original images and the counterfactuals, indicating that Merlin is indeed using appropriate features for image classification.
• Figure 4: Zero-shot cross-modal retrieval. (a) Schematic demonstrating how we perform retrieval. We compute the cosine similarity between Merlin report embeddings and CT embeddings, enabling us to rank CT and report pairs in order of similarity. (b) A distribution of the findings section and impressions section lengths shows that 21% of findings have sequence lengths greater than 512 tokens. (c) Top-1 recall out of pools of 64 findings sections (left), which is considered an in-distribution evaluation as Merlin is trained using findings sections. We also report top-1 recall on out-of-distribution impressions sections (right). (d) An ablation study that examines the impact of using I3D ImageNet initialization, multi-task learning (MTL) versus staged training (Stg.) with EHR and reports versus training with reports only (Rpt.), and splitting the radiology report text into anatomical sections. (e) Data scaling law experiments that examine the impact of pretraining dataset size on retrieval performance. The dashed lines indicate random chance performance.
• Figure 5: Multi-disease 5-year prediction. (a) We fine-tune Merlin for predicting chronic disease onset in otherwise healthy patients within 5-years. (b) We compare Merlin to other baseline model variations fine-tuned for the same task. We find that with both 100% and 10% of downstream training data, Merlin outperforms the other model variations. (c) Comparison of Merlin chronic disease prediction performance to a model trained using only phenotypes (EHR Pretraining), an ImageNet I3D initialized model, and a randomly initialized model. (d) An ablation study that measures the impact of various aspects of Merlin's training strategy. We find that training with EHR and radiology reports, using staged training (Stg.) or multi-task learning (MTL), and training with radiology reports only (Rpt.), all outperform training with EHR only.