A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology

TL;DR

This work tackles the challenge of simultaneous semantic segmentation, instance segmentation, and nuclei classification in digital histology by introducing a bottom-up, all-in-one three-head network that generates semantic maps, edge proposals, and pixel-level class maps. A controlled watershed post-processing step leverages edge and semantic outputs to produce nuclei instances, which are then classified via pixel-grouping against the class maps. The approach yields state-of-the-art performance on PanNuke (Dice 0.841, bPQ 0.713, mPQ 0.633) and generalizes across 19 tissue types with lower model complexity, while ablations confirm the value of the three-head design, weighted losses, and pixel grouping. Despite remaining gaps in dead-nuclei classification and non-neoplastic/neoplastic distinction, the framework significantly improves multi-organ nuclei analysis and offers a practical CAD-ready solution for cancer histology.

Abstract

Simultaneous segmentation and classification of nuclei in digital histology play an essential role in computer-assisted cancer diagnosis; however, it remains challenging. The highest achieved binary and multi-class Panoptic Quality (PQ) remains as low as 0.68 bPQ and 0.49 mPQ, respectively. It is due to the higher staining variability, variability across the tissue, rough clinical conditions, overlapping nuclei, and nuclear class imbalance. The generic deep-learning methods usually rely on end-to-end models, which fail to address these problems associated explicitly with digital histology. In our previous work, DAN-NucNet, we resolved these issues for semantic segmentation with an end-to-end model. This work extends our previous model to simultaneous instance segmentation and classification. We introduce additional decoder heads with independent weighted losses, which produce semantic segmentation, edge proposals, and classification maps. We use the outputs from the three-head model to apply post-processing to produce the final segmentation and classification. Our multi-stage approach utilizes edge proposals and semantic segmentations compared to direct segmentation and classification strategies followed by most state-of-the-art methods. Due to this, we demonstrate a significant performance improvement in producing high-quality instance segmentation and nuclei classification. We have achieved a 0.841 Dice score for semantic segmentation, 0.713 bPQ scores for instance segmentation, and 0.633 mPQ for nuclei classification. Our proposed framework is generalized across 19 types of tissues. Furthermore, the framework is less complex compared to the state-of-the-art.

Figures (10)

• Figure 1: The figure provides an overview of our bottom-up approach framework. The three-head Network produces semantic segmentation, edge proposals, and pixel-wise classes, then using post-processing, instance segmentation and classification are obtained.
• Figure 2: Overview of the three head network architecture and the detailed framework of the sub-blocks. The three head network produce three outputs, i.e., semantic segmentation, edge proposals, and pixel-wise class maps.
• Figure 3: Illustration of the proposed instance segmentation framework.
• Figure 4: Illustration of pixel grouping based on instance segmentation and pixel-level class maps.
• Figure 5: The figure reports the visual results from different organs. The first, second and third row of each organ shows the original image, overlaid predicted semantic segmentation, and the overlaid predicted instances by our framework.