Keep It Accurate and Robust: An Enhanced Nuclei Analysis Framework

TL;DR

This work tackles robust nuclei analysis in histology by introducing a dual-level ensemble framework built on enhanced HoVer-Net variants. By combining intra- and inter-model ensembling with diverse encoder backbones, it achieves superior performance in nuclear instance segmentation, classification, and cellular composition prediction across public benchmarks, notably securing 1st place in composition regression and 3rd in multi-class panoptic quality at CoNIC 2022. The approach is validated on MoNuSAC, PanNuke, and Lizard datasets, with extensive ablations guiding optimal design choices; its practical impact lies in more reliable cellular characterization for cancer diagnosis and prognosis. The authors also release their code and models, providing a scalable toolkit for researchers and clinicians to adopt robust nuclei analysis in diverse clinical settings.

Abstract

Accurate segmentation and classification of nuclei in histology images is critical but challenging due to nuclei heterogeneity, staining variations, and tissue complexity. Existing methods often struggle with limited dataset variability, with patches extracted from similar whole slide images (WSI), making models prone to falling into local optima. Here we propose a new framework to address this limitation and enable robust nuclear analysis. Our method leverages dual-level ensemble modeling to overcome issues stemming from limited dataset variation. Intra-ensembling applies diverse transformations to individual samples, while inter-ensembling combines networks of different scales. We also introduce enhancements to the HoVer-Net architecture, including updated encoders, nested dense decoding and model regularization strategy. We achieve state-of-the-art results on public benchmarks, including 1st place for nuclear composition prediction and 3rd place for segmentation/classification in the 2022 Colon Nuclei Identification and Counting (CoNIC) Challenge. This success validates our approach for accurate histological nuclei analysis. Extensive experiments and ablation studies provide insights into optimal network design choices and training techniques. In conclusion, this work proposes an improved framework advancing the state-of-the-art in nuclei analysis. We release our code and models (https://github.com/WinnieLaugh/CONIC_Pathology_AI) to serve as a toolkit for the community.

Figures (4)

• Figure 1: Overview of the proposed method. A) The base model architecture utilizes an encoder-decoder structure adapted from HoVer-Net graham2019hover. To improve compactness and effectiveness, the three decoder branches have been consolidated into one. Heavy dropout layers are also incorporated in the decoder to regularize training. While any encoder with a similar structure can be used, this example implements a SEResNeXt50 backbone hu2018squeeze. B) The intra-model ensemble approach augments the input image with horizontal and vertical flips. Each base model makes predictions on the original and flipped inputs, which are averaged together after flipping the outputs back to the original orientation. This allows the network to leverage multiple views of the input during inference. C) The inter-model ensemble averages the output maps from base models with different encoder backbones to improve robustness. D) Post-processing utilizes the output maps for final instance recognition. Instances are first segmented using the nuclear presence (NP) and HoVer maps. These segmentation results are then grouped with the nucleus classification (NC) map for instance classification. E) Evaluation uses the multi-class panoptic quality ($mPQ+$) metric applied to the full input patch and multi-class coefficient of determination ($R^2$) applied to the $224\times224$ pixel center region.
• Figure 2: Example nuclei from the 6 categories in the Lizard dataset, illustrating the challenges in segmentation and classification. Note the high degree of similarity among certain classes (e.g., Fig. \ref{['fig:Lizard_dataset']}(b) Epithelial, Fig. \ref{['fig:Lizard_dataset']}(c) Lymphocyte, and Fig. \ref{['fig:Lizard_dataset']}(d) Plasma), which hinders classification.
• Figure 3: Example images from the Lizard dataset. Note the high degree of nuclei overlapping, which poses a significant challenge for instance segmentation methods.
• Figure 4: Qualitative results comparing the proposed method against other state-of-the-art approaches on example images from the Lizard dataset graham2021lizard. The overlays visualize the output instance segmentation results, with different colors indicating the predicted nuclear categories. Examples of nuclei from each category are provided in Fig. \ref{['fig:Lizard_dataset']}. The proposed method produces more accurate predictions than other state-of-the-art techniques, as observed by examining the overlays. Our approach correctly identifies more instances, with fewer false positives and false negatives. Additionally, the nuclear category predictions match the ground truth more closely compared to other methods. This qualitative analysis highlights the improved performance of the proposed framework on this challenging nuclear recognition task.