Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MaskSAM: Towards Auto-prompt SAM with Mask Classification for Volumetric Medical Image Segmentation

Bin Xie, Hao Tang, Bin Duan, Dawen Cai, Yan Yan, Gady Agam

TL;DR

The paper tackles the mismatch between the Segment Anything Model (SAM) and medical image segmentation by introducing MaskSAM, a prompt-free adaptation that enables semantic labeling and volumetric segmentation. It couples a prompt generator with a dataset-mapping pipeline to produce auxiliary masks, boxes, and classifier tokens, and embeds 3D depth-aware adapters (DConvAdapter and DMLPAdapter) within the image encoder and mask decoder to handle volumetric data while keeping SAM's weights largely frozen. Through bipartite matching-based losses and comprehensive evaluations on AMOS2022, ACDC, and Synapse, MaskSAM achieves state-of-the-art Dice scores (e.g., $90.52\%$ on AMOS2022, $93.39\%$ on ACDC, $87.23\%$ on Synapse), outperforming nnUNet and previous SAM-based approaches. The approach demonstrates strong performance with a compact tunable parameter footprint ($14\mathrm{M}$) and offers a practical pathway for plug-and-play, label-aware medical segmentation leveraging large-scale foundation models.

Abstract

Segment Anything Model (SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation, since SAM lacks the ability to predict semantic labels, requires additional prompts, and presents suboptimal performance. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts can solve the requirements of extra prompts. The semantic label prediction can be addressed by the sum of the auxiliary classifier tokens and the learnable global classifier tokens in the mask decoder of SAM. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings to efficiently fine-tune SAM. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.

MaskSAM: Towards Auto-prompt SAM with Mask Classification for Volumetric Medical Image Segmentation

TL;DR

The paper tackles the mismatch between the Segment Anything Model (SAM) and medical image segmentation by introducing MaskSAM, a prompt-free adaptation that enables semantic labeling and volumetric segmentation. It couples a prompt generator with a dataset-mapping pipeline to produce auxiliary masks, boxes, and classifier tokens, and embeds 3D depth-aware adapters (DConvAdapter and DMLPAdapter) within the image encoder and mask decoder to handle volumetric data while keeping SAM's weights largely frozen. Through bipartite matching-based losses and comprehensive evaluations on AMOS2022, ACDC, and Synapse, MaskSAM achieves state-of-the-art Dice scores (e.g., on AMOS2022, on ACDC, on Synapse), outperforming nnUNet and previous SAM-based approaches. The approach demonstrates strong performance with a compact tunable parameter footprint () and offers a practical pathway for plug-and-play, label-aware medical segmentation leveraging large-scale foundation models.

Abstract

Segment Anything Model (SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation, since SAM lacks the ability to predict semantic labels, requires additional prompts, and presents suboptimal performance. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts can solve the requirements of extra prompts. The semantic label prediction can be addressed by the sum of the auxiliary classifier tokens and the learnable global classifier tokens in the mask decoder of SAM. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings to efficiently fine-tune SAM. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.
Paper Structure (14 sections, 1 equation, 7 figures, 5 tables)

This paper contains 14 sections, 1 equation, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. The Proposed Method
  3. Overview of the Proposed MaskSAM
  4. Proposed Dataset Mapping
  5. Proposed Prompt Generator
  6. Proposed Adapters
  7. Modified Image Encoder
  8. Modified Mask Decoder
  9. Losses and Matching
  10. Experiments
  11. Comparison with State-of-the-Art Methods
  12. Ablation Study
  13. Model Analysis
  14. Conclusion

Figures (7)

  • Figure 1: Visualization of predicting auxiliary box prompts. Tending to generate multiple prompts to assist in mask prediction.
  • Figure 2: The overview architecture of our proposed MaskSAM.
  • Figure 3: Overview of (a) redesigned image encoder, (b) proposed prompt generator, and (c) redesigned mask decoder. Blue and white boxes are frozen and the rests are tuned.
  • Figure 4: The proposed adapters.
  • Figure 5: (a) A prompt generator with learnable masks. (b) A prompt generator with learnable boxes. (c) A prompt generator with learnable masks and learnable boxes. (d) A prompt generator with learnable masks and average boxes.
  • ...and 2 more figures