Boosting Medical Image Classification with Segmentation Foundation Model

TL;DR

The paper tackles the challenge of using a Segmentation Foundation Model (SAM) for medical image classification, where naive SAM-based augmentation can hurt performance. It introduces SAMAug-C, which uses segmentation priors to emphasize salient regions, and a two-branch framework that jointly trains on raw and SAMAug-C augmented images, with a weighted ensemble to fuse predictions. Across ISIC 2017, vitiligo, and ExtCRC, SAMAug-C–informed augmentation consistently improves over baselines and outperforms several prior methods, with notable gains for non-pretrained models on ExtCRC. Overall, the work demonstrates that SAM can be repurposed as a medical-image foundation model, enabling improved classification without extensive fine-tuning, through principled augmentation and fusion strategies. Final predictions employ a weighted ensemble: \(Final = \frac{\sum_{i=1}^{2} \omega_i p_i}{2}\) with \(\omega_i \ge 0\) and \(\omega_1+\omega_2=1\).

Abstract

The Segment Anything Model (SAM) exhibits impressive capabilities in zero-shot segmentation for natural images. Recently, SAM has gained a great deal of attention for its applications in medical image segmentation. However, to our best knowledge, no studies have shown how to harness the power of SAM for medical image classification. To fill this gap and make SAM a true ``foundation model'' for medical image analysis, it is highly desirable to customize SAM specifically for medical image classification. In this paper, we introduce SAMAug-C, an innovative augmentation method based on SAM for augmenting classification datasets by generating variants of the original images. The augmented datasets can be used to train a deep learning classification model, thereby boosting the classification performance. Furthermore, we propose a novel framework that simultaneously processes raw and SAMAug-C augmented image input, capitalizing on the complementary information that is offered by both. Experiments on three public datasets validate the effectiveness of our new approach.

Figures (2)

• Figure 1: Visual example from the ISIC 2017 dataset codella2018skin. The entire image augmented by SAMAug zhang2023input is covered in green, which may increase the difficulty for a classifier to distinguish the skin lesion from the background.
• Figure 2: The overview of our framework.