Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing the Power of Prompt-driven Nucleus Instance Segmentation

Zhongyi Shui, Yunlong Zhang, Kai Yao, Chenglu Zhu, Sunyi Zheng, Jingxiong Li, Honglin Li, Yuxuan Sun, Ruizhe Guo, Lin Yang

TL;DR

PromptNucSeg introduces a prompt-driven approach to nucleus instance segmentation in histology images by pairing a nucleus prompter with a fine-tuned Segment Anything Model (SAM). The prompter automatically generates a unique point prompt for each nucleus, aided by an auxiliary nuclear region segmentation task, while adjacent nuclei are used as negative prompts to handle overlaps; a Hungarian matching-based association ensures correct prompt–nucleus pairing. The SAM segmentor is trained with positive prompts per nucleus, enabling one-to-one nucleus mask outputs without heavy post-processing. Across Kumar, CPM-17, and PanNuke, PromptNucSeg achieves state-of-the-art PQ scores and strong nucleus detection/classification with favorable efficiency, though it exhibits higher model size and storage requirements that future work could mitigate through pruning and alternative SAM-like models.

Abstract

Nucleus instance segmentation in histology images is crucial for a broad spectrum of clinical applications. Current dominant algorithms rely on regression of nuclear proxy maps. Distinguishing nucleus instances from the estimated maps requires carefully curated post-processing, which is error-prone and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned huge attention in medical image segmentation, owing to its impressive generalization ability and promptable property. Nevertheless, its potential on nucleus instance segmentation remains largely underexplored. In this paper, we present a novel prompt-driven framework that consists of a nucleus prompter and SAM for automatic nucleus instance segmentation. Specifically, the prompter learns to generate a unique point prompt for each nucleus while the SAM is fine-tuned to output the corresponding mask for the prompted nucleus. Furthermore, we propose the inclusion of adjacent nuclei as negative prompts to enhance the model's capability to identify overlapping nuclei. Without complicated post-processing, our proposed method sets a new state-of-the-art performance on three challenging benchmarks. Code is available at \url{github.com/windygoo/PromptNucSeg}

Unleashing the Power of Prompt-driven Nucleus Instance Segmentation

TL;DR

PromptNucSeg introduces a prompt-driven approach to nucleus instance segmentation in histology images by pairing a nucleus prompter with a fine-tuned Segment Anything Model (SAM). The prompter automatically generates a unique point prompt for each nucleus, aided by an auxiliary nuclear region segmentation task, while adjacent nuclei are used as negative prompts to handle overlaps; a Hungarian matching-based association ensures correct prompt–nucleus pairing. The SAM segmentor is trained with positive prompts per nucleus, enabling one-to-one nucleus mask outputs without heavy post-processing. Across Kumar, CPM-17, and PanNuke, PromptNucSeg achieves state-of-the-art PQ scores and strong nucleus detection/classification with favorable efficiency, though it exhibits higher model size and storage requirements that future work could mitigate through pruning and alternative SAM-like models.

Abstract

Nucleus instance segmentation in histology images is crucial for a broad spectrum of clinical applications. Current dominant algorithms rely on regression of nuclear proxy maps. Distinguishing nucleus instances from the estimated maps requires carefully curated post-processing, which is error-prone and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned huge attention in medical image segmentation, owing to its impressive generalization ability and promptable property. Nevertheless, its potential on nucleus instance segmentation remains largely underexplored. In this paper, we present a novel prompt-driven framework that consists of a nucleus prompter and SAM for automatic nucleus instance segmentation. Specifically, the prompter learns to generate a unique point prompt for each nucleus while the SAM is fine-tuned to output the corresponding mask for the prompted nucleus. Furthermore, we propose the inclusion of adjacent nuclei as negative prompts to enhance the model's capability to identify overlapping nuclei. Without complicated post-processing, our proposed method sets a new state-of-the-art performance on three challenging benchmarks. Code is available at \url{github.com/windygoo/PromptNucSeg}
Paper Structure (27 sections, 7 equations, 4 figures, 8 tables)

This paper contains 27 sections, 7 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Utilization of SAM for Medical Image segmentation
  4. Nucleus Instance Segmentation
  5. Methodology
  6. Preliminaries: SAM
  7. Adapt SAM to nucleus instance segmentation
  8. Learn Prompter
  9. Auxiliary task of nuclear region segmentation
  10. Mask-aided prompt filtering
  11. Use adjacent nuclei as negative prompts
  12. Inference
  13. Experiment
  14. Experiments Settings
  15. Datasets
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Pipeline comparison with currently prevailing nucleus instance segmentation algorithms.
  • Figure 2: (a) The fine-tuning process of SAM. Mask2Prompt signifies randomly sampling a positive point prompt from the foreground area of each nucleus mask. (b) The training procedure of the nucleus prompter. The integration of these two models enables automatic nucleus instance segmentation, as illustrated in Fig. \ref{['fig:banner']} (b).
  • Figure 3: (a) Ground-truth boundary of two overlapping nuclei. (b) Predicted boundary by prompting each nucleus with a positive prompt inside it. (c) Predicted boundary by prompting each nucleus with an additional negative prompt inside its overlapping nucleus. $\mdlgblkcircle$ Positive prompt $\mdlgblkcircle$ Negative prompt
  • Figure 4: Three cases of our method w/ and w/o using negative prompts. The images marked by GT imply the ground-truth nuclear boundaries, while the others indicate predicted outcomes given different types of prompts. $\mdlgblkcircle$ Positive prompt $\mdlgblkcircle$ Negative prompt