HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization

TL;DR

This work tackles the end-to-end segmentation of gigapixel WSIs by introducing HoloHisto, a framework that processes 4K-resolution inputs through sequential tokenization and a linear multi-scale ViT backbone to achieve efficient dense WSI predictions. By enabling direct WSI I/O with online data generation and a 4K random sampler, HoloHisto delivers substantial gains over patch-based baselines and demonstrates robust WSI-level segmentation on a newly curated KPIS dataset of mouse kidney images. The combination of a VQGAN-like sequence tokenizer, three-stage linear attention, and end-to-end WSI inference enables high-fidelity segmentation across ultra-high-resolution images while mitigating computational bottlenecks. This work advances digital pathology by providing a scalable, end-to-end framework and a benchmark dataset that together push toward practical gigapixel WSI analysis.

Abstract

In digital pathology, the traditional method for deep learning-based image segmentation typically involves a two-stage process: initially segmenting high-resolution whole slide images (WSI) into smaller patches (e.g., 256x256, 512x512, 1024x1024) and subsequently reconstructing them to their original scale. This method often struggles to capture the complex details and vast scope of WSIs. In this paper, we propose the holistic histopathology (HoloHisto) segmentation method to achieve end-to-end segmentation on gigapixel WSIs, whose maximum resolution is above 80,000$\times$70,000 pixels. HoloHisto fundamentally shifts the paradigm of WSI segmentation to an end-to-end learning fashion with 1) a large (4K) resolution base patch for elevated visual information inclusion and efficient processing, and 2) a novel sequential tokenization mechanism to properly model the contextual relationships and efficiently model the rich information from the 4K input. To our best knowledge, HoloHisto presents the first holistic approach for gigapixel resolution WSI segmentation, supporting direct I/O of complete WSI and their corresponding gigapixel masks. Under the HoloHisto platform, we unveil a random 4K sampler that transcends ultra-high resolution, delivering 31 and 10 times more pixels than standard 2D and 3D patches, respectively, for advancing computational capabilities. To facilitate efficient 4K resolution dense prediction, we leverage sequential tokenization, utilizing a pre-trained image tokenizer to group image features into a discrete token grid. To assess the performance, our team curated a new kidney pathology image segmentation (KPIs) dataset with WSI-level glomeruli segmentation from whole mouse kidneys. From the results, HoloHisto-4K delivers remarkable performance gains over previous state-of-the-art models.

Figures (4)

• Figure 1: The field of view provided by different resolutions, such as the conventional $512\times512$ patch (indicated by a red box) and $256\times256$, reveals only a limited range of details within the target tissue structures. In contrast, ultra-high resolution (4K) images offer a more comprehensive view of interrelations and can serve as a foundational visual constitute for WSI analysis.
• Figure 2: HoloHisto-4K backbone. To enable scalable encoding of ultra-high-resolution images (4K), our approach uses a pre-trained convolutional VQGAN to learn the from visual parts into discrete tokens, this design follows autoregressive LVM that enables compression while retaining high perception quality. We employed multi-scale linear attention as an efficient way to capture long discrete visual tokens for high-resolution dense prediction.
• Figure 3: The holistic approach for gigapixel WSI segmentation. HoloHisto takes the entire kidney slide image as input, it supports real-time reading of multi-magnification levels and tile sampling. In the inference stage, HoloHisto presents a global gigapixel segmentation mask as output.
• Figure 4: Kidney pathology segmentation qualitative results. We show the comparisons of three different approaches from CNN, ViT-based SAM and our HoloHisto. The right column shows HoloHisto's capability of outputting the entire WSI segmentation.