InstanSeg: an embedding-based instance segmentation algorithm optimized for accurate, efficient and portable cell segmentation

TL;DR

InstanSeg tackles the demand for accurate, fast, and portable cell and nucleus instance segmentation in bioimages. It introduces an embedding-based pipeline that uses a seed-based strategy with positional and conditional embeddings, plus a neural instance head to generate per-pixel probabilities, enabling efficient clustering around seeds. The approach achieves state-of-the-art accuracy on six public datasets while delivering at least a 60% speedup, and it is serialized as TorchScript and exposed via a QuPath extension for broad hardware portability. Open-source Python code and a GUI-friendly QuPath integration facilitate deployment across diverse platforms and workflows.

Abstract

Cell and nucleus segmentation are fundamental tasks for quantitative bioimage analysis. Despite progress in recent years, biologists and other domain experts still require novel algorithms to handle increasingly large and complex real-world datasets. These algorithms must not only achieve state-of-the-art accuracy, but also be optimized for efficiency, portability and user-friendliness. Here, we introduce InstanSeg: a novel embedding-based instance segmentation pipeline designed to identify cells and nuclei in microscopy images. Using six public cell segmentation datasets, we demonstrate that InstanSeg can significantly improve accuracy when compared to the most widely used alternative methods, while reducing the processing time by at least 60%. Furthermore, InstanSeg is designed to be fully serializable as TorchScript and supports GPU acceleration on a range of hardware. We provide an open-source implementation of InstanSeg in Python, in addition to a user-friendly, interactive QuPath extension for inference written in Java. Our code and pre-trained models are available at https://github.com/instanseg/instanseg .

Figures (6)

• Figure 1: The InstanSeg pipeline. A feature encoder $f$ and three feature heads $s,p$ and $e$ transform an input image $\bm{X}$ into a seed map $\bm{S}$, a positional embedding $\bm{P}$ and a conditional embedding $\bm{E}$. We sample local maxima in the seed map to find the coordinates of seed pixels $\bm{u}^k$ (e.g red cross) and compute the relative offsets between the positional embeddings and each seed embedding $\bm{Q}-\bm{Q}^k$. These offsets, along with conditional embeddings $\bm{e}$ serve as input to the instance segmentation head $\Phi$, which outputs a probability map $\bm{\Phi}^k$ of each pixel belonging to instance $k$. The final labelled segmentation map $\bm{\hat{L}}$ is obtained by merging the K probability maps. Input image is from the TNBC 2018 dataset naylor_segmentation_2019.
• Figure 2: InstanSeg uses a local maxima algorithm on a predicted seed map (left image) to locate suitable seed(s) for each object. In large instances, there may be multiple seeds detected within the object. InstanSeg was trained to predict the full instance irrespective of the sampled seed location, as a result predicted objects corresponding to the same instance greatly overlap (centre image). We merge objects with large overlaps by taking their union (right image). Input image from the DSB 2018 dataset caicedo_nucleus_2019.
• Figure 3: Box plot showing the distribution of $F_1^{\mu}$ scores on the entire testing set for each of the six datasets. We compare InstanSeg against each of the three other methods using a paired t-test, and report the significance level of the p-value.
• Figure 4: Qualitative segmentation results across six nucleus segmentation datasets. For each dataset, each of the four methods was retrained from scratch. The displayed images are $128 \times 128$ pixel crops which were randomly selected from the test set.
• Figure 5: InstanSeg inference time is approximately linear with increasing number of detected instances and quadratic with the image dimensions. Timing performed on a laptop GPU. Unlike other methods, the conversion of model outputs to labeled instances is highly efficient, and applying the model backbone (UNet) is the efficiency bottleneck. Timing is for processing 199 images containing 36,073 instances.