Revisiting foundation models for cell instance segmentation

Abstract

Cell segmentation is a fundamental task in microscopy image analysis. Several foundation models for cell segmentation have been introduced, virtually all of them are extensions of Segment Anything Model (SAM), improving it for microscopy data. Recently, SAM2 and SAM3 have been published, further improving and extending the capabilities of general-purpose segmentation foundation models. Here, we comprehensively evaluate foundation models for cell segmentation (CellPoseSAM, CellSAM, $μ$SAM) and for general-purpose segmentation (SAM, SAM2, SAM3) on a diverse set of (light) microscopy datasets, for tasks including cell, nucleus and organoid segmentation. Furthermore, we introduce a new instance segmentation strategy called automatic prompt generation (APG) that can be used to further improve SAM-based microscopy foundation models. APG consistently improves segmentation results for $μ$SAM, which is used as the base model, and is competitive with the state-of-the-art model CellPoseSAM. Moreover, our work provides important lessons for adaptation strategies of SAM-style models to microscopy and provides a strategy for creating even more powerful microscopy foundation models. Our code is publicly available at https://github.com/computational-cell-analytics/micro-sam.

Figures (12)

• Figure 1: a) Overview of our new instance segmentation method, automatic prompt generation (APG), which re-purposes the trained $\mu$SAM (or PathoSAM) models by deriving point prompts from decoder predictions, predicting masks based on these prompts, and then filtering overlapping masks via NMS. Note that the model is not retrained. APG replaces the prior instance segmentation logic. b) Overview of segmentation results for four different microscopy modalities. We report the averaged rank over the 9 datasets per domain in parentheses, top three methods are colored. c) Example label-free cell segmentation with different methods. Only APG correctly segments the large central cell, highlighting its advantage for complex cell morphologies.
• Figure 2: Results for 36 microscopy segmentation datasets in four different modalities: cells (fluorescence, a), cells (label-free, b), nuclei (fluorescence, c), and nuclei (histopathology, d). We indicate the top-3 ranked method with blue colors and methods that were trained on the corresponding training split with textured bars. For our method (APG) we indicate the absolute performance difference with respect to the reference method (AIS).
• Figure 3: Qualitative segmentation results for all microscopy foundation models. We show examples for one dataset per domain / task: cell segmentation in label-free microscopy, cell segmentation in fluorescence microscopy, nucleus segmentation in fluorescence microscopy, and nucleus segmentation in histopathology (top to bottom). Examples for all datasets can be found in Figs. \ref{['fig:app-quali-labelfree']} - \ref{['fig:app-quali-histo']}.
• Figure 4: Comparison of our connected component-based strategy APG (APG (Components)) with an alternative deriving prompts from distance map maxima (APG (Boundary)) for the four different modalities (a-d), reporting the relative mean segmentation accuracy difference w.r.t AIS.
• Figure 5: Qualitative results for all label-free microscopy datasets for cell instance segmentation.