Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

Junde Wu, Wei Ji, Yuanpei Liu, Huazhu Fu, Min Xu, Yanwu Xu, Yueming Jin

TL;DR

This work tackles the gap in applying a general segmentation model to medical imaging by introducing the Medical SAM Adapter (Med-SA). It combines Space-Depth Transpose (SD-Trans) for 2D-to-3D adaptation and a Hyper-Prompting Adapter (HyP-Adpt) for prompt-conditioned adaptation, while freezing most of SAM and updating only ~2% of parameters. Across 17 medical segmentation tasks and multiple modalities, Med-SA outperforms SAM, fully fine-tuned MedSAM, and several SOTA methods, including Swin-UNetr, often with substantial parameter efficiency. The results demonstrate strong cross-domain transferability and practical potential for interactive, foundation-model-based medical image segmentation, with code released for reproducibility.

Abstract

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.

Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

TL;DR

This work tackles the gap in applying a general segmentation model to medical imaging by introducing the Medical SAM Adapter (Med-SA). It combines Space-Depth Transpose (SD-Trans) for 2D-to-3D adaptation and a Hyper-Prompting Adapter (HyP-Adpt) for prompt-conditioned adaptation, while freezing most of SAM and updating only ~2% of parameters. Across 17 medical segmentation tasks and multiple modalities, Med-SA outperforms SAM, fully fine-tuned MedSAM, and several SOTA methods, including Swin-UNetr, often with substantial parameter efficiency. The results demonstrate strong cross-domain transferability and practical potential for interactive, foundation-model-based medical image segmentation, with code released for reproducibility.

Abstract

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.
Paper Structure (17 sections, 2 equations, 4 figures, 3 tables)

This paper contains 17 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interactive Segmentation
  4. Parameter-Efficient Fine-Turning
  5. Method
  6. Preliminary: SAM architecture
  7. Med-SA architecture
  8. SD-Trans
  9. HyP-Adpt
  10. Training Strategy
  11. Experiments
  12. Dataset
  13. Implementation Details
  14. Comparing with SOTA on Abdominal Multi-organ Segmentation
  15. Comparing with SOTA on Multi-modality Images
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Med-SA architecture. We use (b) as the encoder with standard Adapter to process 2D medical images, and (c) incorporating SD-Trans to process 3D images. Then we use (d) as the decoder with HyP-Adpt to incorporate the prompts.
  • Figure 2: HyP-Adpt architecture. We utilize Prompt Embedding to generate the weights that are applied to the Adapter Embedding.
  • Figure 3: Visual comparison of Med-SA and SAM on abdominal multi-organ segmentation. We use Check mark to represent SAM correctly found the organ and Cross to represent it lost.
  • Figure 4: Visual comparison of Med-SA and SAM on medical image segmentation with four different modalities. Top-left: optic disc and cup segmentation from the fundus image. Top-right: brain tumor segmentation from the Brain MRI. Bottom-left: melanoma segmentation from the dermoscopic image. Bottom-right: thyroid nodule segmentation from the ultrasound image.