CellPilot: A unified approach to automatic and interactive segmentation in histopathology

TL;DR

CellPilot tackles the need for reliable segmentation in histopathology by unifying automatic and interactive approaches. It combines a bounding-box–driven CellViT module with a finetuned SAM, enabling automatic segmentation that users can refine with positive or negative prompts, trained on over 675,000 masks across nine datasets. The method demonstrates superior interactive performance on held-out datasets compared with existing tools, and is released with an open-source GUI to support large-scale annotation. By balancing automation with user control, the approach points to future improvements such as semantic classification and efficiency optimization.

Abstract

Histopathology, the microscopic study of diseased tissue, is increasingly digitized, enabling improved visualization and streamlined workflows. An important task in histopathology is the segmentation of cells and glands, essential for determining shape and frequencies that can serve as indicators of disease. Deep learning tools are widely used in histopathology. However, variability in tissue appearance and cell morphology presents challenges for achieving reliable segmentation, often requiring manual correction to improve accuracy. This work introduces CellPilot, a framework that bridges the gap between automatic and interactive segmentation by providing initial automatic segmentation as well as guided interactive refinement. Our model was trained on over 675,000 masks of nine diverse cell and gland segmentation datasets, spanning 16 organs. CellPilot demonstrates superior performance compared to other interactive tools on three held-out histopathological datasets while enabling automatic segmentation. We make the model and a graphical user interface designed to assist practitioners in creating large-scale annotated datasets available as open-source, fostering the development of more robust and generalized diagnostic models.

Figures (3)

• Figure 1: CellPilot allows interactive refinement of automated predictions. We fine-tune SAM (a) with simulated automated interactive refinements (b) on a total of nine diverse datasets spanning 16 organs and over 675,000 masks of cells and glands and test it on three held-out test sets (c). During inference, the framework works in two stages: In the first part, CellViT cellvit is used to provide bounding boxes, which are used as input to the prompt encoder to produce automatic predictions. In the interactive stage, the user provides points or bounding boxes to add new masks or refine existing masks using positive (enlarging segmentation area) or negative prompts (decreasing it).
• Figure 2: CellPilot outperforms MedSAM, SimpleClick and SAM on the cell segmentation datasets MoNuSAC and CellSeg, as well as on the gland segmentation dataset CRAG. We compare performance when starting with a point prompt (left) and box prompt (right) and, step by step, automatically adding points in regions of wrong prediction, simulating interactive refinement. Note that SimpleClick does not support box prompts.
• Figure 3: CellPilot produces coherent masks. Qualitative comparison of samples from CellSeg, MoNuSAC, and CRAG datasets using point prompts (left) or box prompts (right).