CellGenNet: A Knowledge-Distilled Framework for Robust Cell Segmentation in Cancer Tissues

TL;DR

This work tackles robust nuclei segmentation in heterogeneous cancer WSIs under limited annotations. It introduces CellGenNet, a semi-supervised knowledge-distillation framework where a StarDist teacher generates pseudo-labels to guide a lightweight U-Net student, guided by a bias-correcting compound loss and consistency regularization. The approach yields superior Dice, IoU, and boundary accuracy compared with StarDist, Cellpose, and InstanSeg on Osteosarcoma and NuInsSeg datasets, notably improving performance on irregular and densely clustered nuclei. By reducing the need for extensive manual annotations while maintaining cross-tissue generalization, CellGenNet enables scalable and reproducible histopathology analysis across diverse cancer tissues.

Abstract

Accurate nuclei segmentation in microscopy whole slide images (WSIs) remains challenging due to variability in staining, imaging conditions, and tissue morphology. We propose CellGenNet, a knowledge distillation framework for robust cross-tissue cell segmentation under limited supervision. CellGenNet adopts a student-teacher architecture, where a capacity teacher is trained on sparse annotations and generates soft pseudo-labels for unlabeled regions. The student is optimized using a joint objective that integrates ground-truth labels, teacher-derived probabilistic targets, and a hybrid loss function combining binary cross-entropy and Tversky loss, enabling asymmetric penalties to mitigate class imbalance and better preserve minority nuclear structures. Consistency regularization and layerwise dropout further stabilize feature representations and promote reliable feature transfer. Experiments across diverse cancer tissue WSIs show that CellGenNet improves segmentation accuracy and generalization over supervised and semi-supervised baselines, supporting scalable and reproducible histopathology analysis.

Figures (3)

• Figure 1: An overview of CellGenNet student–teacher segmentation framework.
• Figure 2: Qualitative results of the compared segmentation model outputs against the ground truth for the Osteosarcoma and NuInsSeg Datasets. The segmentation boundary are overlaid onto the images (blue color). Best viewed in zoomed mode.
• Figure 3: Box plots showing the performance of all methods on both datasets and significance levels using a Mann-Whitney U Test. A $p \leq 0.001$ is denoted by three stars "***", a $p \leq 0.01$ is denoted by two stars "**", a $p \leq 0.05$ is denoted by one star "*", and a $p > 0.5$ is denoted by "ns".