Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models

Mobina Mansoori, Sajjad Shahabodini, Farnoush Bayatmakou, Jamshid Abouei, Konstantinos N. Plataniotis, Arash Mohammadi

TL;DR

The paper addresses the challenge of limited labeled data and domain shift in medical image analysis by evaluating seven state-of-the-art natural-domain foundation models (AIMv2, DINOv2, SAM2, MAE, VMamba, CoCa, CLIP) when fine-tuned on four medical datasets (CBIS-DDSM, ISIC2019, APTOS2019, CHEXPERT). It compares frozen versus unfrozen backbones and linear versus multi-head attention classifiers, across varying backbone sizes. The results show AIMv2, DINOv2, and SAM2 deliver the strongest performance, with dataset size influencing the optimal backbone and architecture; multi-head attention provides advantages in frozen settings, while full fine-tuning yields gains across models. The findings suggest that advances in natural-domain training generalize to medical imaging, enabling robust classification with limited labels, and the authors provide open-source code for reproducibility.

Abstract

Using massive datasets, foundation models are large-scale, pre-trained models that perform a wide range of tasks. These models have shown consistently improved results with the introduction of new methods. It is crucial to analyze how these trends impact the medical field and determine whether these advancements can drive meaningful change. This study investigates the application of recent state-of-the-art foundation models, DINOv2, MAE, VMamba, CoCa, SAM2, and AIMv2, for medical image classification. We explore their effectiveness on datasets including CBIS-DDSM for mammography, ISIC2019 for skin lesions, APTOS2019 for diabetic retinopathy, and CHEXPERT for chest radiographs. By fine-tuning these models and evaluating their configurations, we aim to understand the potential of these advancements in medical image classification. The results indicate that these advanced models significantly enhance classification outcomes, demonstrating robust performance despite limited labeled data. Based on our results, AIMv2, DINOv2, and SAM2 models outperformed others, demonstrating that progress in natural domain training has positively impacted the medical domain and improved classification outcomes. Our code is publicly available at: https://github.com/sajjad-sh33/Medical-Transfer-Learning.

Advancements in Medical Image Classification through Fine-Tuning Natural Domain Foundation Models

TL;DR

The paper addresses the challenge of limited labeled data and domain shift in medical image analysis by evaluating seven state-of-the-art natural-domain foundation models (AIMv2, DINOv2, SAM2, MAE, VMamba, CoCa, CLIP) when fine-tuned on four medical datasets (CBIS-DDSM, ISIC2019, APTOS2019, CHEXPERT). It compares frozen versus unfrozen backbones and linear versus multi-head attention classifiers, across varying backbone sizes. The results show AIMv2, DINOv2, and SAM2 deliver the strongest performance, with dataset size influencing the optimal backbone and architecture; multi-head attention provides advantages in frozen settings, while full fine-tuning yields gains across models. The findings suggest that advances in natural-domain training generalize to medical imaging, enabling robust classification with limited labels, and the authors provide open-source code for reproducibility.

Abstract

Using massive datasets, foundation models are large-scale, pre-trained models that perform a wide range of tasks. These models have shown consistently improved results with the introduction of new methods. It is crucial to analyze how these trends impact the medical field and determine whether these advancements can drive meaningful change. This study investigates the application of recent state-of-the-art foundation models, DINOv2, MAE, VMamba, CoCa, SAM2, and AIMv2, for medical image classification. We explore their effectiveness on datasets including CBIS-DDSM for mammography, ISIC2019 for skin lesions, APTOS2019 for diabetic retinopathy, and CHEXPERT for chest radiographs. By fine-tuning these models and evaluating their configurations, we aim to understand the potential of these advancements in medical image classification. The results indicate that these advanced models significantly enhance classification outcomes, demonstrating robust performance despite limited labeled data. Based on our results, AIMv2, DINOv2, and SAM2 models outperformed others, demonstrating that progress in natural domain training has positively impacted the medical domain and improved classification outcomes. Our code is publicly available at: https://github.com/sajjad-sh33/Medical-Transfer-Learning.

Paper Structure

This paper contains 11 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. METHODOLOGY
  3. Foundation Models
  4. Datasets and Evaluation metrics
  5. EXPERIMENTS
  6. Implementation Details
  7. Model Performance Comparison
  8. Linear and Multi Attention Head
  9. Model Size
  10. Training Time
  11. Conclusion

Figures (1)

  • Figure 1: Comparison of different sizes of backbones in each foundation model across all datasets, based on the linear head of the unfreeze configuration.