CISCA and CytoDArk0: a Cell Instance Segmentation and Classification method for histo(patho)logical image Analyses and a new, open, Nissl-stained dataset for brain cytoarchitecture studies

TL;DR

This work tackles automatic cell instance segmentation and classification in histology, introducing CISCA-Net, a lightweight U-Net-based architecture with three heads for boundary, distance-map regression, and cell-type classification. A novel four-direction distance map design and a boundary-focused pixel-classification scheme enable robust separation of touching cells, complemented by a multi-component loss and a tailored post-processing pipeline. The authors also release CytoDArk0, the first open Nissl-stained brain dataset for cytoarchitecture studies, totaling nearly 40k annotated cells across brain regions and species. Across CoNIC, PanNuke, MoNuSeg, and CytoDArk0, CISCA achieves competitive or state-of-the-art performance with a small parameter footprint, demonstrating robustness to staining, magnification, and tissue type, and enabling detailed cell morphology and counting in digital pathology and comparative neuroanatomy.

Abstract

Delineating and classifying individual cells in microscopy tissue images is inherently challenging yet remains essential for advancements in medical and neuroscientific research. In this work, we propose a new deep learning framework, CISCA, for automatic cell instance segmentation and classification in histological slices. At the core of CISCA is a network architecture featuring a lightweight U-Net with three heads in the decoder. The first head classifies pixels into boundaries between neighboring cells, cell bodies, and background, while the second head regresses four distance maps along four directions. The outputs from the first and second heads are integrated through a tailored post-processing step, which ultimately produces the segmentation of individual cells. The third head enables the simultaneous classification of cells into relevant classes, if required. We demonstrate the effectiveness of our method using four datasets, including CoNIC, PanNuke, and MoNuSeg, which are publicly available H&Estained datasets that cover diverse tissue types and magnifications. In addition, we introduce CytoDArk0, the first annotated dataset of Nissl-stained histological images of the mammalian brain, containing nearly 40k annotated neurons and glia cells, aimed at facilitating advancements in digital neuropathology and brain cytoarchitecture studies. We evaluate CISCA against other state-of-the-art methods, demonstrating its versatility, robustness, and accuracy in segmenting and classifying cells across diverse tissue types, magnifications, and staining techniques. This makes CISCA well-suited for detailed analyses of cell morphology and efficient cell counting in both digital pathology workflows and brain cytoarchitecture research.

Figures (8)

• Figure 1: Illustration of relevant maps for the instance segmentation of cells a) Sample patch from the CytoDArk0 dataset. b) Instance segmentation label map (GT) c) Semantic segmentation map - $3$ pixel classes with all-round contours. d) Semantic segmentation map - $3$ pixel classes with the contours class limited to the boundary between cells that touch or are in close proximity. e) Example of a distance map along a specific direction as in Hover-Net graham2019hover or in our proposed approach, CISCA.
• Figure 2: An overview of the overall CISCA inference pipeline for cell instance segmentation and classification on WSIs. An image of any size is divided into overlapping patches. Each patch is processed individually by CISCA-Net. The outputs are merged during an untiling step. Finally, a tailored post-processing is applied to generate the instance segmentation map and assign a cell type to each detected cell (cf. Section \ref{['sec:post']}).
• Figure 3: A diagram of the architecture of the suggested CISCA-Net for cell instance segmentation and classification. CISCA-Net is a DL architecture featuring a lightweight U-Net backbone and three convolutional heads. The first head (yellow) focuses on classifying pixels into boundary between closely located cells (green), cell bodies (pink), and background (white). The second head (purple) handles the regression of four oriented distance maps, predicting distances from the cell centroid for each pixel. Distance maps are normalized to increase from a minimum of $-1$ (purple) to a maximum of $1$ (green). The third head (blue) classifies pixels into different cell types, depending on the dataset.
• Figure 4: Effect of the augmentation strategy on sample patches from the CoNIC (first row), PanNuke (second row), CytoDArk0_20x_256 (third row), and CytoDArk0_40x_1024 (fourth row) datasets. The original patch is shown in the leftmost position of a row and is followed on the right by patches obtained by applying random transformations as detailed in Section \ref{['sec:StainNormalizationAugmentation']}.
• Figure 5: Top: Morphological and density properties of cells in different brain areas from CytoDArk0. Bottom: Sample images from CytoDArk0. In particular, images are taken from CytoDArk0_20x_512, a subset of CytoDArk0 with images at $20$x and size $512\times512$.