APSeg: Auto-Prompt Model with Acquired and Injected Knowledge for Nuclear Instance Segmentation and Classification

TL;DR

APSeg tackles the dependency on precise prompts in SAM for nuclear instance segmentation and classification by introducing an auto-prompter with two knowledge-driven modules. DG-POM learns nuclear distribution via a density-map counting task to deform sampling proposals, while CK-SIM injects morphology-aware knowledge through a CLIP-based class-aware query, improving localization and category discrimination. Evaluations on PanNuke and CoNSeP demonstrate state-of-the-art or competitive performance across segmentation and classification metrics, with ablations confirming the complementary benefits of DG-POM and CK-SIM. This approach offers a practical, knowledge-enhanced prompt generation pipeline that enhances nuclei analysis in digital pathology and reduces reliance on handcrafted prompts.

Abstract

Nuclear instance segmentation and classification provide critical quantitative foundations for digital pathology diagnosis. With the advent of the foundational Segment Anything Model (SAM), the accuracy and efficiency of nuclear segmentation have improved significantly. However, SAM imposes a strong reliance on precise prompts, and its class-agnostic design renders its classification results entirely dependent on the provided prompts. Therefore, we focus on generating prompts with more accurate localization and classification and propose \textbf{APSeg}, \textbf{A}uto-\textbf{P}rompt model with acquired and injected knowledge for nuclear instance \textbf{Seg}mentation and classification. APSeg incorporates two knowledge-aware modules: (1) Distribution-Guided Proposal Offset Module (\textbf{DG-POM}), which learns distribution knowledge through density map guided, and (2) Category Knowledge Semantic Injection Module (\textbf{CK-SIM}), which injects morphological knowledge derived from category descriptions. We conducted extensive experiments on the PanNuke and CoNSeP datasets, demonstrating the effectiveness of our approach. The code will be released upon acceptance.

Figures (3)

• Figure 1: Overview of APSeg. (a) APSeg consists of two knowledge-aware modules DG-POM and CK-SIM, with a (b) point-to-instance SAM for instance segmentation followed the prompt generated by APSeg.
• Figure 2: The detailed architectures of (a) Distribution Guided Proposal Offset module (DG-POM) and (b) Category Knowledge Semantic Injection Module (CK-SIM). The parts of knowledge acquisition and injection are highlighted with pink lines.
• Figure 3: The visual comparison of ablation study for CK-SIM(a) and DG-POM(b).