Adapting Segment Anything Model to Melanoma Segmentation in Microscopy Slide Images

TL;DR

A novel approach that uses the Segment Anything Model (SAM) for automatic melanoma segmentation in microscopy slide images and designs a dynamic prompting strategy that uses a combination of centroid and grid prompts to achieve optimal coverage of the super high-resolution slide images while maintaining the quality of generated prompts.

Abstract

Melanoma segmentation in Whole Slide Images (WSIs) is useful for prognosis and the measurement of crucial prognostic factors such as Breslow depth and primary invasive tumor size. In this paper, we present a novel approach that uses the Segment Anything Model (SAM) for automatic melanoma segmentation in microscopy slide images. Our method employs an initial semantic segmentation model to generate preliminary segmentation masks that are then used to prompt SAM. We design a dynamic prompting strategy that uses a combination of centroid and grid prompts to achieve optimal coverage of the super high-resolution slide images while maintaining the quality of generated prompts. To optimize for invasive melanoma segmentation, we further refine the prompt generation process by implementing in-situ melanoma detection and low-confidence region filtering. We select Segformer as the initial segmentation model and EfficientSAM as the segment anything model for parameter-efficient fine-tuning. Our experimental results demonstrate that this approach not only surpasses other state-of-the-art melanoma segmentation methods but also significantly outperforms the baseline Segformer by 9.1% in terms of IoU.

Figures (5)

• Figure 1: An overview of the proposed method. The initial mask $X$ generated by Segformer is post-processed to generate the mask $\hat{X}^m$. We run SAM on the prompts generated from $\hat{X}^m$ to generate the mask $\hat{Y}^m$. The two masks are combined to create the final mask Y.
• Figure 2: In-situ melanoma detection finds the estimated in-situ melanoma regions $X^s$ from the initial mask $X$. Low-confidence region filtering discards low-confidence connected components from $X^s$.
• Figure 3: Prompt Generation Strategy. We apply the centroid prompt and the grid prompt to based on the geometric characteristics of each connected component.
• Figure 4: Qualitative results on our dataset. Red denotes invasive melanoma and green denotes the epidermis. Compared to Segformer, our method significantly improves the accuracy of predictions for invasive melanoma regions, especially in areas where distinguishing from the epidermis is challenging.
• Figure : Determine prompt type for a connected component