RepSNet: A Nucleus Instance Segmentation model based on Boundary Regression and Structural Re-parameterization

TL;DR

RepSNet presents a nucleus instance segmentation framework grounded in nucleus boundary regression and a Boundary Voting Mechanism, encapsulated in a fully re-parameterizable encoder-decoder network. By predicting boundary distance maps and aggregating them across nucleus pixels, it achieves robust boundary delineation even in densely overlapping nuclei, while structural re-parameterization improves inference efficiency. The NBRIA scheme, along with a bespoke loss L_nb based on nucleus boundary isoheights, yields superior PQ/AJI/DICE metrics on the Lizard dataset, and demonstrates transferable gains when integrated with other segmentation models. The work offers a practical, scalable approach for accurate nucleus segmentation and classification with potential extensions to broader medical and imaging domains.

Abstract

Pathological diagnosis is the gold standard for tumor diagnosis, and nucleus instance segmentation is a key step in digital pathology analysis and pathological diagnosis. However, the computational efficiency of the model and the treatment of overlapping targets are the major challenges in the studies of this problem. To this end, a neural network model RepSNet was designed based on a nucleus boundary regression and a structural re-parameterization scheme for segmenting and classifying the nuclei in H\&E-stained histopathological images. First, RepSNet estimates the boundary position information (BPI) of the parent nucleus for each pixel. The BPI estimation incorporates the local information of the pixel and the contextual information of the parent nucleus. Then, the nucleus boundary is estimated by aggregating the BPIs from a series of pixels using a proposed boundary voting mechanism (BVM), and the instance segmentation results are computed from the estimated nucleus boundary using a connected component analysis procedure. The BVM intrinsically achieves a kind of synergistic belief enhancement among the BPIs from various pixels. Therefore, different from the methods available in literature that obtain nucleus boundaries based on a direct pixel recognition scheme, RepSNet computes its boundary decisions based on some guidances from macroscopic information using an integration mechanism. In addition, RepSNet employs a re-parametrizable encoder-decoder structure. This model can not only aggregate features from some receptive fields with various scales which helps segmentation accuracy improvement, but also reduce the parameter amount and computational burdens in the model inference phase through the structural re-parameterization technique. Extensive experiments demonstrated the superiorities of RepSNet compared to several typical benchmark models.

Figures (7)

• Figure 1: Challenges in the nucleus instance segmentation for histopathological images: a widespread adhesion of nuclei (in the cases in the first row), multiple species, shape variety, and low contrast between the nucleus foreground and the cytoplasmic background (in the cases in the second row). Three H&E stained images, their instance segmentation ground truth, and predictions from the proposed method are repectively presented from left to right. In the images of the first column, nuclei are presented in dark color; In the images of the second column and the third column, various non-black colors are used to indicate different nuclei.
• Figure 2: Three types of nucleus instance segmentation methods: (a) Pixel Discrimination (PD) method, (b) Nucleus Centroid Detection and Information Propagation (NCDIP) method, and (c) Nucleus Boundary Regression and Information Aggregation (NBRIA) method (our proposed method). The PD method obtains the nucleus foreground and boundary predictions based on some pixel-by-pixel classification schemes, and separates the overlapping nucleus instances based on the above-mentioned predictions. The NCDIP method first estimates the probability of each pixel belonging to one nucleus and extracts the centroid area (CA) of every nucleus using the predicted probability; Then, the NCDIP computes the outmost area for each nucleus and the corresponding instance segmentation result by propagating the information from the CA based on the local feature similarities of some neighbor pixels. NBRIA describes the nucleus instance segmentation as a boundary regression problem: NBRIA estimates the boundary position information (BPI) of the parent nucleus in four direction from each nucleus pixel using a proposed RepSNet, computes the nucleus boundary estimation by aggregating the BPIs from a series of pixels using a proposed boundary voting mechanism (BVM), and obtains the instance segmentation results by a connected component analysis procedure.
• Figure 3: A flowchart of the proposed NBRIA for nucleus instance segmentation and recognition. For an input image, the proposed RepSNet computes three direct outputs: a nucleus pixel (NP) feature map, a nucleus type (NT) feature map, and a boundary distance (BD) feature map. The BD feature map is a matrix consisting of the distances from each pixel to the boundary of its parent nucleus in four directions. Furtherly, RepSNet estimates a nucleus boundary (NB) feature map from the NP feature map and BD feature map using a proposed Boundary Voting Mechanism (BVM, section \ref{['subsubsec:bvm']}). This BVM procedure fuses the boundary position information (BPI) estimated from each pixel of a nucleus. Therefore, RepSNet systematically exploits local information and global contextual information in determining the nucleus boundary. Then, the NBRIA computes a nucleus instance (NI) feature map from the NB feature map using a connectivity analysis procedure. Finally, the type of each instance is determined by the most frequent occurring type in that instance area.
• Figure 4: The network architecture of the proposed RepSNet. (a) RepSNet is a U-Net-like re-parameterizable neural network model with a shared encoder, and two structurally identical and mutually independent decoders. One decoder is used to determine whether a pixel comes from a nucleus, and which category of the parent nucleus (seven categories of nuclei in the experiments of this work). The other decoder is a regression branch for estimating the distances from each pixel to the boundary of the parent nucleus in four directions (up, down, left, and right). (b) RepVgg Unit Ding2021RepVGG is a feature extraction module with structural re-parameterization functions. In the training phase, the RepVgg Unit has three convolutional branches with different-size receptive fields to improve the sensitiveness of the model to multi-scale features. In the inference phase, three branches of the RepVgg Unit are fused into one convolutional branch by a structural re-parameterization technique to reduce the parameter amount and the computation burdens of the model. (c) RepUpsample is a proposed upsampling module with a structural re-parameterization technique. In literature, the structural re-parameterization technique is used in some downsampling procedures.
• Figure 5: Two sketch maps for computing the nucleus boundary loss ($L_{nb}$). (a) A sketch map of nucleus instance segmentation. (b) A sketch map of the nucleus boundary isoheights. The closer the isoheight deviates from the labeled nucleus boundary, the smaller the energy of nucleus boundary loss; The farther the isoheight deviates from the labeled nucleus boundary, the larger the energy of nucleus boundary loss. This boundary isoheight feature map can be used to approximate the closest distance between the estimated target boundary and the labeled boundary. Based on the approximated distance, we can penalize the deviations of the boundary estimations adaptively and efficiently.