Cell Nuclei Detection and Classification in Whole Slide Images with Transformers

TL;DR

CellNuc-DETR reframes nuclei analysis in whole slide images as direct detection and classification using a DETR-based transformer approach, achieving state-of-the-art accuracy while markedly improving inference speed over segmentation-based methods. The method employs a Swin-based backbone with multi-scale deformable attention and an efficient in-device sliding-window pipeline to handle gigapixel WSIs. Across PanNuke, CoNSeP, and MoNuSeg, it demonstrates strong generalization and robustness, validated by cross-dataset evaluations and fast WSI inference on TCGA slides. Compared with segmentation-centric baselines such as HoVer-NeXt and CellViT, CellNuc-DETR delivers superior or comparable accuracy with substantially lower post-processing overhead, enabling high-throughput clinical and research workflows in digital pathology.

Abstract

Accurate and efficient cell nuclei detection and classification in histopathological Whole Slide Images (WSIs) are pivotal for digital pathology applications. Traditional cell segmentation approaches, while commonly used, are computationally expensive and require extensive post-processing, limiting their practicality for high-throughput clinical settings. In this paper, we propose a paradigm shift from segmentation to detection for extracting cell information from WSIs, introducing CellNuc-DETR as a more effective solution. We evaluate the accuracy performance of CellNuc-DETR on the PanNuke dataset and conduct cross-dataset evaluations on CoNSeP and MoNuSeg to assess robustness and generalization capabilities. Our results demonstrate state-of-the-art performance in both cell nuclei detection and classification tasks. Additionally, we assess the efficiency of CellNuc-DETR on large WSIs, showing that it not only outperforms current methods in accuracy but also significantly reduces inference times. Specifically, CellNuc-DETR is twice as fast as the fastest segmentation-based method, HoVer-NeXt, while achieving substantially higher accuracy. Moreover, it surpasses CellViT in accuracy and is approximately ten times more efficient in inference speed on WSIs. These results establish CellNuc-DETR as a superior approach for cell analysis in digital pathology, combining high accuracy with computational efficiency.

Figures (12)

• Figure 1: Cell nuclei class distribution across tissues on the PanNuke dataset.
• Figure 2: CellNuc-DETR model architecture.
• Figure 3: CellNuc-DETR inference pipeline on WSIs.
• Figure 4: Resolution of edge cells between overlapped windows.
• Figure 5: PanNuke ground truth and CellNuc-DETR$_{tiny,3lvl}$ predictions across tissues.