Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adapting a Segmentation Foundation Model for Medical Image Classification

Pengfei Gu, Haoteng Tang, Islam A. Ebeid, Jose A. Nunez, Fabian Vazquez, Diego Adame, Marcus Zhan, Huimin Li, Bin Fu, Danny Z. Chen

TL;DR

The paper tackles the challenge of leveraging segmentation foundation models for medical image classification by freezing the SAM image encoder to extract segmentation-rich features and introducing Spatially Localized Channel Attention (SLCA) to fuse these features into standard DL classifiers. A two-branch architecture combines a frozen SAM feature extractor with a DL classifier, using SLCA to generate spatially aware channel weights and a projector head to enrich fusion. Empirical results across RetinaMNIST, BreastMNIST, and ISIC 2017 show data-efficient improvements in accuracy and AUC, with SLCA and the projector head yielding meaningful gains and outperforming the SAMAug-C baseline. The work offers a practical, data-efficient approach to incorporate segmentation priors into medical imaging workflows without extensive fine-tuning.

Abstract

Recent advancements in foundation models, such as the Segment Anything Model (SAM), have shown strong performance in various vision tasks, particularly image segmentation, due to their impressive zero-shot segmentation capabilities. However, effectively adapting such models for medical image classification is still a less explored topic. In this paper, we introduce a new framework to adapt SAM for medical image classification. First, we utilize the SAM image encoder as a feature extractor to capture segmentation-based features that convey important spatial and contextual details of the image, while freezing its weights to avoid unnecessary overhead during training. Next, we propose a novel Spatially Localized Channel Attention (SLCA) mechanism to compute spatially localized attention weights for the feature maps. The features extracted from SAM's image encoder are processed through SLCA to compute attention weights, which are then integrated into deep learning classification models to enhance their focus on spatially relevant or meaningful regions of the image, thus improving classification performance. Experimental results on three public medical image classification datasets demonstrate the effectiveness and data-efficiency of our approach.

Adapting a Segmentation Foundation Model for Medical Image Classification

TL;DR

The paper tackles the challenge of leveraging segmentation foundation models for medical image classification by freezing the SAM image encoder to extract segmentation-rich features and introducing Spatially Localized Channel Attention (SLCA) to fuse these features into standard DL classifiers. A two-branch architecture combines a frozen SAM feature extractor with a DL classifier, using SLCA to generate spatially aware channel weights and a projector head to enrich fusion. Empirical results across RetinaMNIST, BreastMNIST, and ISIC 2017 show data-efficient improvements in accuracy and AUC, with SLCA and the projector head yielding meaningful gains and outperforming the SAMAug-C baseline. The work offers a practical, data-efficient approach to incorporate segmentation priors into medical imaging workflows without extensive fine-tuning.

Abstract

Recent advancements in foundation models, such as the Segment Anything Model (SAM), have shown strong performance in various vision tasks, particularly image segmentation, due to their impressive zero-shot segmentation capabilities. However, effectively adapting such models for medical image classification is still a less explored topic. In this paper, we introduce a new framework to adapt SAM for medical image classification. First, we utilize the SAM image encoder as a feature extractor to capture segmentation-based features that convey important spatial and contextual details of the image, while freezing its weights to avoid unnecessary overhead during training. Next, we propose a novel Spatially Localized Channel Attention (SLCA) mechanism to compute spatially localized attention weights for the feature maps. The features extracted from SAM's image encoder are processed through SLCA to compute attention weights, which are then integrated into deep learning classification models to enhance their focus on spatially relevant or meaningful regions of the image, thus improving classification performance. Experimental results on three public medical image classification datasets demonstrate the effectiveness and data-efficiency of our approach.
Paper Structure (8 sections, 1 equation, 3 figures, 4 tables)

This paper contains 8 sections, 1 equation, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. SAM Image Encoder: A Frozen Feature Extractor
  4. Spatially Localized Channel Attention (SLCA)
  5. Adapting SAM Image Encoder for Medical Image Classification
  6. Experiments and Results
  7. Conclusions
  8. Acknowledgement

Figures (3)

  • Figure 1: An overview of our framework. In $[C, H, W]$, $C$ represents the number of channels, and $H$ and $W$ represent the height and width of a feature map, respectively. For simplicity, the BN and ReLU that follow each Conv layer are not shown.
  • Figure 2: Visual examples of different attention methods on the ISIC 2017 (top row) and BreastMNIST (bottom row) datasets.
  • Figure 3: Visual result demonstrating that meaningful semantics are extracted by the SAM Image Encoder on the ISIC 2017 dataset: (a) the input image, and (b) the heat map of feature extracted by the SAM Image Encoder.