Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST

Fuping Wu, Bartlomiej W. Papiez

TL;DR

This work provides a comprehensive benchmark of diverse foundation models on MedMNIST for medical image classification, comparing end-to-end fine-tuning and linear probing across CNN and ViT backbones. It reveals that ViT-based models typically deliver superior transfer performance, with end-to-end training outperforming linear probing in most cases, though data efficiency and pipeline differences can alter conclusions. The study highlights the importance of image resizing strategies and data volume, and finds that medical-data pretraining does not uniformly outperform non-medical baselines, signaling a need for more robust medical foundation-model development. Collectively, the results offer practical guidance for model selection, hyperparameter tuning, and preprocessing in medical image classification tasks, while outlining avenues for broader dataset validation and deeper bias analyses.

Abstract

Foundation models are widely employed in medical image analysis, due to their high adaptability and generalizability for downstream tasks. With the increasing number of foundation models being released, model selection has become an important issue. In this work, we study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset. Specifically, we adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks. The results demonstrate the significant potential of these pre-trained models when transferred for medical image classification. We further conduct experiments with different image sizes and various sizes of training data. By analyzing all the results, we provide preliminary, yet useful insights and conclusions on this topic.

Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST

TL;DR

This work provides a comprehensive benchmark of diverse foundation models on MedMNIST for medical image classification, comparing end-to-end fine-tuning and linear probing across CNN and ViT backbones. It reveals that ViT-based models typically deliver superior transfer performance, with end-to-end training outperforming linear probing in most cases, though data efficiency and pipeline differences can alter conclusions. The study highlights the importance of image resizing strategies and data volume, and finds that medical-data pretraining does not uniformly outperform non-medical baselines, signaling a need for more robust medical foundation-model development. Collectively, the results offer practical guidance for model selection, hyperparameter tuning, and preprocessing in medical image classification tasks, while outlining avenues for broader dataset validation and deeper bias analyses.

Abstract

Foundation models are widely employed in medical image analysis, due to their high adaptability and generalizability for downstream tasks. With the increasing number of foundation models being released, model selection has become an important issue. In this work, we study the capabilities of foundation models in medical image classification tasks by conducting a benchmark study on the MedMNIST dataset. Specifically, we adopt various foundation models ranging from convolutional to Transformer-based models and implement both end-to-end training and linear probing for all classification tasks. The results demonstrate the significant potential of these pre-trained models when transferred for medical image classification. We further conduct experiments with different image sizes and various sizes of training data. By analyzing all the results, we provide preliminary, yet useful insights and conclusions on this topic.
Paper Structure (22 sections, 34 figures, 7 tables)

This paper contains 22 sections, 34 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Pre-trained Model Selection
  4. General Validation Framework
  5. Benchmark Results
  6. Datasets
  7. Results
  8. Learning rate selection for end-to-end fine-tuning strategy
  9. End-to-end fine-tuning vs linear probing
  10. CNN models vs ViT-based models
  11. Impact of training pipeline
  12. Discussion
  13. Effect of Image Size and Resizing Strategy
  14. Data Efficiency
  15. Transferring Foundation Models Pre-trained on Medical Data
  16. ...and 7 more sections

Figures (34)

  • Figure 1: Framework of our study. We evaluate the performance using the MedMNIST dataset collection and select foundation models from a representative pool.
  • Figure 2: Accuracy of DINO ViT-B/16 on DermaMNIST with various numbers of training data for each class, and "full" means using all training data.
  • Figure 3: Comparing the Accuracy of VGG16 with the learning rate of the encoder ranging in $\{10^{-3},10^{-4},10^{-5}\}$ on different datasets.
  • Figure 4: Comparing the Accuracy of ResNet-18 with the learning rate of the encoder ranging in $\{10^{-3},10^{-4},10^{-5}\}$ on different datasets.
  • Figure 5: Comparing the Accuracy of DenseNet-121 with the learning rate of the encoder ranging in $\{10^{-3},10^{-4},10^{-5}\}$ on different datasets.
  • ...and 29 more figures