Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

Stefano Woerner, Christian F. Baumgartner

TL;DR

This study tackles data scarcity in medical imaging by benchmarking few-shot and zero-shot learning using 16 foundation models across 19 MedIMeta datasets. It combines supervised, self-supervised, and CLIP-based models with linear probing and selective fine-tuning to assess performance under realistic computation constraints, including prompt-based ZSL. The results show BiomedCLIP excels in ultra-low data settings, while large CLIP variants like CLIP-ViT-H outperform with more labeled data; zero-shot learning lags behind FSL in medical tasks. Overall, the findings underscore the potential of medical-domain foundation models, the importance of dataset diversity, and the need for further medical-focused pretraining to advance practical clinical deployment.

Abstract

Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. Although sufficient data exists for some well-studied medical tasks, there remains a long tail of clinically relevant tasks with poor data availability. Recently, numerous foundation models have demonstrated high suitability for few-shot learning (FSL) and zero-shot learning (ZSL), potentially making them more accessible to practitioners. However, it remains unclear which foundation model performs best on FSL medical image analysis tasks and what the optimal methods are for learning from limited data. We conducted a comprehensive benchmark study of ZSL and FSL using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes, while very large CLIP models pretrained on LAION-2B perform best with slightly more training samples. However, simply fine-tuning a ResNet-18 pretrained on ImageNet performs similarly with more than five training examples per class. Our findings also highlight the need for further research on foundation models specifically tailored for medical applications and the collection of more datasets to train these models.

Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging

TL;DR

This study tackles data scarcity in medical imaging by benchmarking few-shot and zero-shot learning using 16 foundation models across 19 MedIMeta datasets. It combines supervised, self-supervised, and CLIP-based models with linear probing and selective fine-tuning to assess performance under realistic computation constraints, including prompt-based ZSL. The results show BiomedCLIP excels in ultra-low data settings, while large CLIP variants like CLIP-ViT-H outperform with more labeled data; zero-shot learning lags behind FSL in medical tasks. Overall, the findings underscore the potential of medical-domain foundation models, the importance of dataset diversity, and the need for further medical-focused pretraining to advance practical clinical deployment.

Abstract

Data scarcity is a major limiting factor for applying modern machine learning techniques to clinical tasks. Although sufficient data exists for some well-studied medical tasks, there remains a long tail of clinically relevant tasks with poor data availability. Recently, numerous foundation models have demonstrated high suitability for few-shot learning (FSL) and zero-shot learning (ZSL), potentially making them more accessible to practitioners. However, it remains unclear which foundation model performs best on FSL medical image analysis tasks and what the optimal methods are for learning from limited data. We conducted a comprehensive benchmark study of ZSL and FSL using 16 pretrained foundation models on 19 diverse medical imaging datasets. Our results indicate that BiomedCLIP, a model pretrained exclusively on medical data, performs best on average for very small training set sizes, while very large CLIP models pretrained on LAION-2B perform best with slightly more training samples. However, simply fine-tuning a ResNet-18 pretrained on ImageNet performs similarly with more than five training examples per class. Our findings also highlight the need for further research on foundation models specifically tailored for medical applications and the collection of more datasets to train these models.
Paper Structure (13 sections, 5 figures, 1 table)

This paper contains 13 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Dataset
  4. Simulation of FSL and ZSL tasks
  5. Pretrained Models
  6. Few-shot Learning Strategies
  7. Zero-shot learning strategy
  8. Metrics
  9. Experiments and Results
  10. Conclusion
  11. Acknowledgments.
  12. Disclosure of Interests.
  13. Supplementary Materials

Figures (5)

  • Figure 1: Harmonic mean AUROC over all 19 MedIMeta datasets
  • Figure 2: AUROC on the different datasets with fully supervised baseline from woerner2024comprehensive. The fully-supervised performance is indicated by the black dotted line.
  • Figure 3: Harmonic mean AUROC across all 19 MedIMeta datasets of fine-tuned ResNet models with the best-performing linear probe as a point of comparison.
  • Figure 4: Model properties plotted against mean few-shot performance over the tested $n$. The symbol indicates the type of pretraining (Supervised, DINO, CLIP).
  • Figure A.1: Harmonic mean AUROC across all 19 datasets with three zero-shot templates.