BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis

TL;DR

The paper investigates scaling laws for self-supervised biomedical vision foundation models, addressing how model size, data diversity, imaging modalities, and training objectives affect performance in medical imaging. It introduces BioVFM-21M, a 21-million-image biomedical dataset, and BioVFM, a 630M-parameter vision foundation model pretrained on this data, evaluated across 12 MedMNIST benchmarks using linear probing and two SSL methods (MAE and DINO V2). A key finding is that scaling benefits are task- and modality-dependent, with data diversity and pretraining objectives strongly influencing outcomes, while simply increasing data size yields diminishing returns; scalability is also shaped by data distribution complexity as captured by metrics like DBI and g-zip compressibility, and the scaling slope $a$ from $y = e^b x^a$. The work demonstrates state-of-the-art performance on medical benchmarks and provides actionable guidelines for constructing scalable medical vision models, emphasizing data diversity, modality coverage, and efficient pretraining; the BioVFM-21M dataset and BioVFM model are intended to be released publicly to spur further advances in biomedical AI.

Abstract

Scaling up model and data size have demonstrated impressive performance improvement over a wide range of tasks. Despite extensive studies on scaling behaviors for general-purpose tasks, medical images exhibit substantial differences from natural data. It remains unclear the key factors in developing medical vision foundation models at scale due to the absence of an extensive understanding of scaling behavior in the medical domain. In this paper, we explored the scaling behavior across model sizes, training algorithms, data sizes, and imaging modalities in developing scalable medical vision foundation models by self-supervised learning. To support scalable pretraining, we introduce BioVFM-21M, a large-scale biomedical image dataset encompassing a wide range of biomedical image modalities and anatomies. We observed that scaling up does provide benefits but varies across tasks. Additional analysis reveals several factors correlated with scaling benefits. Finally, we propose BioVFM, a large-scale medical vision foundation model pretrained on 21 million biomedical images, which outperforms the previous state-of-the-art foundation models across 12 medical benchmarks. Our results highlight that while scaling up is beneficial for pursuing better performance, task characteristics, data diversity, pretraining methods, and computational efficiency remain critical considerations for developing scalable medical foundation models.

Figures (4)

• Figure 1: The proposed BioVFM-21M dataset.a. BioVFM-21M covers a wide range of imaging modalities and anatomical areas. The illustration is exponential-scaled. b. Comparison with existing large-scale biomedical image datasets. c. BioVFM-21M covers images from microcosmic to macroscopic.
• Figure 2: Medical FMs scales with model size, data size, and number of modalities.a. Model scaling of BioVFM-D and BioVFM-M. The dotted line indicates the fitted scaling curves and the value within brackets indicates the slope $a$. The turquoise star indicates the performance of BiomedCLIP. b. Data and modality scaling with BioVFM-D. The results are averaged over 12 medical benchmarks.
• Figure 3: Model scaling results across 12 medical benchmarks. The dotted line indicates the fitted scaling curves and the value within brackets indicates the slope $a$. The turquoise star indicates the performance of BiomedCLIP.
• Figure 4: Model scaling slopes and correlation to benchmark properties.a. The model scaling slopes of BioVFM-D with 12 medical benchmarks. b. The model scaling slopes of BioVFM-M with 12 medical benchmarks. The colors are consistent for each task. c. The Pearson correlation coefficient between model scaling slopes and benchmark metrics.