Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets

TL;DR

This study tests the zero-shot Segment Anything Model (SAM) on 12 public medical image segmentation datasets (7,451 subjects) and compares its performance to five domain-specific models. It systematically evaluates three SAM prompt modes (Semantic, Point, Box), analyzes how 6 factors (including segmentation difficulty and modality) affect accuracy, and uses Dice overlap as the primary metric. The findings show SAM underperforms relative to specialized models across all datasets, with 2D images and easier segmentation tasks yielding higher Dice, and identify target size and segmentation difficulty as key determinants. The work highlights the need to adapt SAM to medical imaging through domain-specific fine-tuning or new benchmark models, while acknowledging limitations in 3D handling and prompting granularity.

Abstract

Background: The segment-anything model (SAM), introduced in April 2023, shows promise as a benchmark model and a universal solution to segment various natural images. It comes without previously-required re-training or fine-tuning specific to each new dataset. Purpose: To test SAM's accuracy in various medical image segmentation tasks and investigate potential factors that may affect its accuracy in medical images. Methods: SAM was tested on 12 public medical image segmentation datasets involving 7,451 subjects. The accuracy was measured by the Dice overlap between the algorithm-segmented and ground-truth masks. SAM was compared with five state-of-the-art algorithms specifically designed for medical image segmentation tasks. Associations of SAM's accuracy with six factors were computed, independently and jointly, including segmentation difficulties as measured by segmentation ability score and by Dice overlap in U-Net, image dimension, size of the target region, image modality, and contrast. Results: The Dice overlaps from SAM were significantly lower than the five medical-image-based algorithms in all 12 medical image segmentation datasets, by a margin of 0.1-0.5 and even 0.6-0.7 Dice. SAM-Semantic was significantly associated with medical image segmentation difficulty and the image modality, and SAM-Point and SAM-Box were significantly associated with image segmentation difficulty, image dimension, target region size, and target-vs-background contrast. All these 3 variations of SAM were more accurate in 2D medical images, larger target region sizes, easier cases with a higher Segmentation Ability score and higher U-Net Dice, and higher foreground-background contrast.

Figures (4)

• Figure 1: The framework of the proposed method. Different medical images are collected from 11 different organs and 6 different modalities. We compared two typical models for medical image segmentation: state-of-the-art segmentation networks trained on medical images and SAM trained on more than 1 million natural images.
• Figure 2: Three variations of SAM (SAM-Semantic, SAM-Point, SAM-Box) as tested in this paper. We used a liver tumor CT image as an example.
• Figure 3: Accuracy of SAM in 12 medical image segmentation datasets. (a): Dice overlaps of 5 medical-image-specific algorithms and 3 variations of SAM. (b): scatter plots of SAM's Dice (y axis) with U-Net's Dice (x axis) across datasets, for SAM-Semantic (left), SAM-Point (middle), and SAM-Box (right).
• Figure 4: Single-factor analysis associating 6 potential factors with SAM's accuracies in 12 medical image datasets. In the top four rows, every dot is a subject, and the color of the dot denotes the dataset this subject comes from. p-values are provided and highlighted with red background if a factor has a significant association with the SAM Dice overlap. In the $6^{th}$ row, Derm--Dermology; Colo--Colonoscopy; US--Ultrasound.