NucFuseRank: Dataset Fusion and Performance Ranking for Nuclei Instance Segmentation

TL;DR

Nuclei instance segmentation in H&E histology suffers from dataset bias and heterogeneous annotation standards, complicating fair model benchmarking. The paper introduces NucFuse, a standardized benchmark comprising a unified test set (NucFuse-test) and a fused training set (NucFuse-train), built from 10 publicly available datasets and evaluated with two state-of-the-art models, HoVerNeXt and CellViT, across single-dataset and fused-training regimes. It provides a dataset ranking based on cross-dataset generalization, demonstrates the benefits of dataset fusion (notably excluding CryoNuSeg due to domain shift), and releases code and datasets for reproducible benchmarking. The findings show that dataset quality and diversity significantly shape performance, with merged training data yielding substantial gains and offering a practical benchmark to guide dataset selection and evaluation for nuclei instance segmentation in histology. $PQ$ and related metrics are used to quantify performance, supporting a rigorous, cross-dataset evaluation standard that can accelerate development of robust histopathology analysis tools.

Abstract

Nuclei instance segmentation in hematoxylin and eosin (H&E)-stained images plays an important role in automated histological image analysis, with various applications in downstream tasks. While several machine learning and deep learning approaches have been proposed for nuclei instance segmentation, most research in this field focuses on developing new segmentation algorithms and benchmarking them on a limited number of arbitrarily selected public datasets. In this work, rather than focusing on model development, we focused on the datasets used for this task. Based on an extensive literature review, we identified manually annotated, publicly available datasets of H&E-stained images for nuclei instance segmentation and standardized them into a unified input and annotation format. Using two state-of-the-art segmentation models, one based on convolutional neural networks (CNNs) and one based on a hybrid CNN and vision transformer architecture, we systematically evaluated and ranked these datasets based on their nuclei instance segmentation performance. Furthermore, we proposed a unified test set (NucFuse-test) for fair cross-dataset evaluation and a unified training set (NucFuse-train) for improved segmentation performance by merging images from multiple datasets. By evaluating and ranking the datasets, performing comprehensive analyses, generating fused datasets, conducting external validation, and making our implementation publicly available, we provided a new benchmark for training, testing, and evaluating nuclei instance segmentation models on H&E-stained histological images.

Figures (9)

• Figure 1: An overview of the entire workflow. First, a unified test set (NucFuse-test) was constructed from all datasets to evaluate the experiments. The remaining images were then used to train two state-of-the-art models, CellViT HORST2024103143 and HoVerNeXt baumann2024hover, under two experimental setups: single-dataset training (e.g., training only on PCNS or only on NuInsSeg) and fused-dataset training (progressively merging the training datasets). Based on the results of Experiment 1, the datasets were ranked according to their performance on the unified test set. Based on the results of Experiment 2, an optimal merged training dataset (NucFuse-train) was introduced.
• Figure 2: Results from Single-Dataset Training. For each training dataset, HoVerNeXt and CellViT were trained using only the corresponding dataset, and their instance segmentation performance based on panoptic quality (PQ) score was evaluated separately on each constituent subset of the unified test set (NucFuse-test).
• Figure 3: Dataset cross-correlation matrices for HoVerNeXt (a) and CellViT (b) models.
• Figure 4: Top three performing training datasets for each test subset for HoVerNeXt (top) and CellViT (bottom).
• Figure 5: Results of merging the top-K best datasets and training the models, evaluated on the NucFuse-test dataset based on panoptic quality (PQ) score (%).