CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models

TL;DR

CLIMB presents a large-scale, multimodal clinical benchmark that unifies 4.51 million samples across 44 public datasets and 15 modalities (imaging, text, time series, graphs) to accelerate holistic foundation models in healthcare. Through extensive multitask pretraining, the study shows improved performance on understudied modalities and better generalization to novel tasks, with gains up to $32.54\%$ in ultrasound AUC and notable improvements in ECG. The paper demonstrates strong few-shot transfer and reveals that unimodal encoders pretrained on CLIMB can boost multimodal fusion performance, with cross-attention fusion excelling for complex tasks like length of stay, and simpler fusion often sufficing for other outcomes. By releasing code and pretrained weights, CLIMB offers a scalable, adaptable data foundation for future clinical AI research while highlighting architecture- and fusion-level design choices and potential societal considerations around privacy and fairness.

Abstract

Recent advances in clinical AI have enabled remarkable progress across many clinical domains. However, existing benchmarks and models are primarily limited to a small set of modalities and tasks, which hinders the development of large-scale multimodal methods that can make holistic assessments of patient health and well-being. To bridge this gap, we introduce Clinical Large-Scale Integrative Multimodal Benchmark (CLIMB), a comprehensive clinical benchmark unifying diverse clinical data across imaging, language, temporal, and graph modalities. CLIMB comprises 4.51 million patient samples totaling 19.01 terabytes distributed across 2D imaging, 3D video, time series, graphs, and multimodal data. Through extensive empirical evaluation, we demonstrate that multitask pretraining significantly improves performance on understudied domains, achieving up to 29% improvement in ultrasound and 23% in ECG analysis over single-task learning. Pretraining on CLIMB also effectively improves models' generalization capability to new tasks, and strong unimodal encoder performance translates well to multimodal performance when paired with task-appropriate fusion strategies. Our findings provide a foundation for new architecture designs and pretraining strategies to advance clinical AI research. Code is released at https://github.com/DDVD233/climb.

Figures (7)

• Figure 1: Overview of the CLIMB framework for training and testing multimodal datasets. The training phase incorporates diverse data modalities: graphs (molecular, BrainNet), 1D signals (ECG/EEG, genomics, EHR), 2D images (X-rays, dermoscopy, fundus, mammograms, pathology), and 3D scans (CT, MRI, endoscopy, ultrasound). Through multitask training on heterogeneous clinical data, our framework enhances model performance across individual tasks, particularly for understudied modalities defined in Fig. \ref{['fig:figure_of_figures']}(b). This approach improves both generalization to novel tasks and multimodal understanding when combined with appropriate fusion strategies, ultimately advancing performance on critical clinical applications like disease diagnosis and patient risk prediction.
• Figure 2: Overview of CLIMB benchmark and code.(a) Visualization of CLIMB dataset composition. The inner ring displays the primary data modalities (2D, 1D, Graph, Multimodal). The middle ring represents major clinical modalities within each modality. The outer ring shows the names of specific datasets within each category, with the outer bar plot representing the number of samples in that dataset. A detailed description of each modality and datasets are included in App. \ref{['sec:individual_dataset_detials']}. (b) Focus of dataset collection. During the collection of CLIMB, we aim to collect a diverse range of datasets, with a special focus on novel tasks, datasets from understudied modalities, and datasets from underrepresented regions. (c) Distribution of data collection sites in CLIMB. Red regions indicate areas where clinical datasets are commonly collected, whereas blue regions indicate places where clinical dataset collections are rare. (d) Example code usage on CLIMB framework. This code example loads a custom mixed subset of CLIMB spanning across three modalities, then trains a ConvNextv2 classifier on the dataset mixture with unified vocabulary. (e) Sample data from CLIMB. CLIMB preserves detailed labels, metadata and comments explaining the diagnosis.
• Figure 3: Experimental setup for evaluating (a) multitask, (b) transfer, and (c) fusion learning strategies, addressing RQ1, 2, 3 respectively. (a) investigates how multitask pretrained clinical models work across multiple tasks consistently, especially understudied tasks. (b) explores how well multitask pretrained clinical models transfer to new tasks with few samples within the same clinical modality. (c) experiments on whether multitask pretrained unimodal models can be fused effectively to tackle multimodal clinical tasks.
• Figure 4: Difference in AUC achieved by the multitask model compared to single-task training.Novel Task represent emerging clinical challenges like COVID-19; Underrepresented Regions indicates datasets from underrepresented regions in developing countries such as Brazil and China; Understudied Modalities includes less common imaging types such as ultrasound and CT scans. Datasets belonging to multiple categories are highlighted in pink. In general, multitask learning helps the model to reach a better performance, with the greatest improvement observed in understudied tasks, regions, and/or modalities.
• Figure 5: Few-shot learning performance comparison across different time series domains. We evaluate the few-shot performance (1-shot, 8-shot, and full dataset) of three representative models: BIOT for EEGs, ECG-JEPA and UniTS for ECGs. PT on Standard shows the performance when pretrained on their datasets from the original paper, while PT on CLIMB shows the performance when pretrained on our CLIMB dataset. Models pretrained on CLIMB demonstrate consistent improvements over the original ECG domain-specific models.