Learning to Generalize over Subpartitions for Heterogeneity-aware Domain Adaptive Nuclei Segmentation

TL;DR

This work tackles nuclei instance segmentation under annotation scarcity and modality/stain shifts by introducing open compound domain adaptation (OCDA) with a two-stage disentanglement framework. Stage I performs cross-domain image translation with image-level content/style disentanglement and progressive subdomain clustering to reveal cancer-type-specific styles, complemented by a nucleus-shape-preserving translation. Stage II enforces instance-level domain-invariant representations via a Mask R-CNN backbone, augmented with global-local style consistency and subdomain-aware style regularization to improve cross-domain performance. Together, these components deliver substantial improvements over state-of-the-art UDA and OCDA methods across cross-modality and cross-stain scenarios, with strong generalization to unseen subdomains and no requirement for target-domain labels.

Abstract

Annotation scarcity and cross-modality/stain data distribution shifts are two major obstacles hindering the application of deep learning models for nuclei analysis, which holds a broad spectrum of potential applications in digital pathology. Recently, unsupervised domain adaptation (UDA) methods have been proposed to mitigate the distributional gap between different imaging modalities for unsupervised nuclei segmentation in histopathology images. However, existing UDA methods are built upon the assumption that data distributions within each domain should be uniform. Based on the over-simplified supposition, they propose to align the histopathology target domain with the source domain integrally, neglecting severe intra-domain discrepancy over subpartitions incurred by mixed cancer types and sampling organs. In this paper, for the first time, we propose to explicitly consider the heterogeneity within the histopathology domain and introduce open compound domain adaptation (OCDA) to resolve the crux. In specific, a two-stage disentanglement framework is proposed to acquire domain-invariant feature representations at both image and instance levels. The holistic design addresses the limitations of existing OCDA approaches which struggle to capture instance-wise variations. Two regularization strategies are specifically devised herein to leverage the rich subpartition-specific characteristics in histopathology images and facilitate subdomain decomposition. Moreover, we propose a dual-branch nucleus shape and structure preserving module to prevent nucleus over-generation and deformation in the synthesized images. Experimental results on both cross-modality and cross-stain scenarios over a broad range of diverse datasets demonstrate the superiority of our method compared with state-of-the-art UDA and OCDA methods.

Figures (16)

• Figure 1: Examples of histopathology images and cropped regions of different cancer types from the Kumar dataset Kumar2017. From left to right: liver cancer, kidney cancer, and colon cancer.
• Figure 2: Illustration of the OCDA setting in a benchmark performing domain adaptation from fluorescence microscopy to histopathology images. Note that, unlike multi-target UDA Saporta2021, the cancer type of each image patch is unavailable during training.
• Figure 3: Overview of Stage I for the proposed two-stage framework. The main objective of Stage I is to mitigate the significant image appearance discrepancy between different modalities and staining techniques with cross-domain image translation. A DRIT Lee2020drit-like architecture is employed as backbone with several auxiliary modules to overcome its limitations.
• Figure 4: Illustration of the progressive clustering and separation strategy. In this module, we enforce intra-subdomain style compactness as well as inter-subdomain style separation to benefit feature disentanglement. Considering that the pseudo subdomain labels are highly noisy, especially in the early training stage, we only compute losses based on reliable samples which have high confidence for clustering results. As the style encoder is gradually trained, more samples will become reliable and consequently the style encodings of all image patches will form clear cluster organizations.
• Figure 5: Overview of Stage II for the proposed two-stage framework. In this stage, the inputs are the target-like source images synthesized with model trained in Stage I. Afterward, cross-domain feature alignment is performed via a Mask RCNN-based instance-level feature disentanglement network for domain adaptive instance segmentation.