SAM on Medical Images: A Comprehensive Study on Three Prompt Modes

TL;DR

The paper evaluates whether SAM can function as a zero-shot foundation for medical image segmentation by comparing auto-, box-, and point-prompt modes across 12 multimodal datasets. It finds that box-prompting with zero jitter provides the strongest average performance, while auto-prompt underperforms and increasing the number of prompt points in point-prompt mode helps close the gap to box-prompt results. A key insight is that box-jitter significantly degrades accuracy, underscoring the sensitivity of SAM prompts to bounding-box quality. Overall, SAM offers useful zero-shot segmentation signals in medical imaging but generally falls short of supervised, task-tuned baselines, highlighting the importance of prompt design and future improvement directions.

Abstract

The Segment Anything Model (SAM) made an eye-catching debut recently and inspired many researchers to explore its potential and limitation in terms of zero-shot generalization capability. As the first promptable foundation model for segmentation tasks, it was trained on a large dataset with an unprecedented number of images and annotations. This large-scale dataset and its promptable nature endow the model with strong zero-shot generalization. Although the SAM has shown competitive performance on several datasets, we still want to investigate its zero-shot generalization on medical images. As we know, the acquisition of medical image annotation usually requires a lot of effort from professional practitioners. Therefore, if there exists a foundation model that can give high-quality mask prediction simply based on a few point prompts, this model will undoubtedly become the game changer for medical image analysis. To evaluate whether SAM has the potential to become the foundation model for medical image segmentation tasks, we collected more than 12 public medical image datasets that cover various organs and modalities. We also explore what kind of prompt can lead to the best zero-shot performance with different modalities. Furthermore, we find that a pattern shows that the perturbation of the box size will significantly change the prediction accuracy. Finally, Extensive experiments show that the predicted mask quality varied a lot among different datasets. And providing proper prompts, such as bounding boxes, to the SAM will significantly increase its performance.

Figures (3)

• Figure 1: Segmentation results with different prompt modes
• Figure 2: We show the pattern of dice changing with different level of jitter over most of the datasets.
• Figure 3: Segmentation results with different prompt modes