Instance-Aware Robust Consistency Regularization for Semi-Supervised Nuclei Instance Segmentation

TL;DR

This work tackles nuclei instance segmentation under limited annotations by introducing IRCR-Net, a semi-supervised framework that enforces instance-level consistency through two mechanisms: Matching-Driven Instance-Aware Consistency (MIAC) and Prior-Driven Instance-Aware Consistency (PIAC). MIAC aligns teacher and student instance proposals via bipartite matching using a centroid-based distance and only enforces loss on matched pairs, while PIAC leverages KDE-derived priors from external datasets to filter pseudo-labels, reducing noise. The combined approach, within a Mean-Teacher architecture and a watershed-based instance segmentation module, yields strong improvements over baselines across MoNuSeg, MoNuSAC, PanNuke, and CoNSeP, often matching or exceeding fully supervised performance under limited labeled data. The results demonstrate robust, instance-level regularization that better handles dense and overlapping nuclei, with practical implications for pathology analytics and tumor microenvironment studies.

Abstract

Nuclei instance segmentation in pathological images is crucial for downstream tasks such as tumor microenvironment analysis. However, the high cost and scarcity of annotated data limit the applicability of fully supervised methods, while existing semi-supervised methods fail to adequately regularize consistency at the instance level, lack leverage of the inherent prior knowledge of pathological structures, and are prone to introducing noisy pseudo-labels during training. In this paper, we propose an Instance-Aware Robust Consistency Regularization Network (IRCR-Net) for accurate instance-level nuclei segmentation. Specifically, we introduce the Matching-Driven Instance-Aware Consistency (MIAC) and Prior-Driven Instance-Aware Consistency (PIAC) mechanisms to refine the nuclei instance segmentation result of the teacher and student subnetwork, particularly for densely distributed and overlapping nuclei. We incorporate morphological prior knowledge of nuclei in pathological images and utilize these priors to assess the quality of pseudo-labels generated from unlabeled data. Low-quality pseudo-labels are discarded, while high-quality predictions are enhanced to reduce pseudo-label noise and benefit the network's robust training. Experimental results demonstrate that the proposed method significantly enhances semi-supervised nuclei instance segmentation performance across multiple public datasets compared to existing approaches, even surpassing fully supervised methods in some scenarios.

Figures (7)

• Figure 1: Holistic consistency aligns global maps and thus admits merge/split errors and error injection via previous EMA.
• Figure 2: Overview of the proposed Instance-Aware Robust Consistency Regularization Network (IRCR-Net) for semi-supervised nuclei instance segmentation. The framework incorporates a teacher-student network architecture with two key consistency mechanisms, Matching-Driven Instance-Aware Consistency (MIAC) and Prior-Driven Instance-Aware Consistency (PIAC), to improve semi-supervised nuclei instance segmentation performance.
• Figure 3: Overview of the prior-driven instance selection process.
• Figure 4: Comparison with other methods on MoNuSAC datasets under varying labeled training data ratios.
• Figure 5: Visual comparison of nuclei segmentation results from different methods: (a) Input image, (b) Ground truth, (c) Hover-Net, (d) ST, (e) MT, (f) PG-FANet, (g) Ours, and (h) FullSup.