Scanner-Induced Domain Shifts Undermine the Robustness of Pathology Foundation Models

TL;DR

Pathology foundation models (PFMs) are powerful encoders for whole-slide images, but their robustness to scanner-induced domain shifts is unclear. The authors establish a controlled multiscanner benchmark using the CHIME Multiscanner dataset and TCGA-BRCA to evaluate 14 PFMs across unsupervised embedding geometry and supervised clinicopathological tasks, revealing that most PFMs exhibit scanner-specific embedding shifts and downstream calibration biases even when AUC remains high. They show that robustness does not scale with model size or recency; vision-language models like CONCH offer some embedding stability, and robustness-oriented distillation (H0-mini) improves cross-scanner consistency but may worsen calibration. The study argues for embedding-stability and calibration-focused evaluation in PFMs, highlighting practical implications for clinical deployment and guiding future robustness-oriented model design and validation.

Abstract

Pathology foundation models (PFMs) have become central to computational pathology, aiming to offer general encoders for feature extraction from whole-slide images (WSIs). Despite strong benchmark performance, PFM robustness to real-world technical domain shifts, such as variability from whole-slide scanner devices, remains poorly understood. We systematically evaluated the robustness of 14 PFMs to scanner-induced variability, including state-of-the-art models, earlier self-supervised models, and a baseline trained on natural images. Using a multiscanner dataset of 384 breast cancer WSIs scanned on five devices, we isolated scanner effects independently from biological and laboratory confounders. Robustness is assessed via complementary unsupervised embedding analyses and a set of clinicopathological supervised prediction tasks. Our results demonstrate that current PFMs are not invariant to scanner-induced domain shifts. Most models encode pronounced scanner-specific variability in their embedding spaces. While AUC often remains stable, this masks a critical failure mode: scanner variability systematically alters the embedding space and impacts calibration of downstream model predictions, resulting in scanner-dependent bias that can impact reliability in clinical use cases. We further show that robustness is not a simple function of training data scale, model size, or model recency. None of the models provided reliable robustness against scanner-induced variability. While the models trained on the most diverse data, here represented by vision-language models, appear to have an advantage with respect to robustness, they underperformed on downstream supervised tasks. We conclude that development and evaluation of PFMs requires moving beyond accuracy-centric benchmarks toward explicit evaluation and optimisation of embedding stability and calibration under realistic acquisition variability.

Figures (15)

• Figure 1: Overview of the study design for evaluating scanner-induced domain shifts in pathology foundation models (PFMs). A1) A total of 384 breast cancer whole-slide images (WSIs) from as many patients were scanned on five different whole-slide scanner devices, forming the CHIME Multiscanner dataset. Each physical slide was scanned using all five devices, thereby isolating scanner-induced variability from all other sources. A2) WSIs from the CHIME Multiscanner and TCGA-BRCA datasets were preprocessed using a standardised workflow and encoded using 14 frozen feature extractors $f_{\theta_i}$, comprising 13 PFMs and a ResNet baseline trained on natural images. B) Scanner-variability robustness is evaluated through complementary qualitative and quantitative unsupervised analyses of feature embedding geometry, capturing both global structure and local neighbourhood consistency across scanners. C) Supervised downstream benchmarking is performed to further assess clinical relevance, with models trained on TCGA-BRCA and evaluated on CHIME Multiscanner. This setup enables systematic evaluation of scanner-induced effects on predictive performance, prediction consistency and calibration stability.
• Figure 2: Summary of dataset composition for CHIME Multiscanner and TCGA-BRCA. Donut charts illustrate the class distribution for the clinical biomarkers (ER, PR, HER2) and histological grades (NHG1, NHG2, NHG3) within the training (TCGA-BRCA) and evaluation (CHIME Multiscanner) cohorts. For CHIME Multiscanner, the proportion of labeled and unlabeled WSIs is also shown, as unlabeled data is utilized for the unsupervised embedding analysis.
• Figure 3: Representative mosaic of tissue tiles sampled from unique WSIs in the CHIME Multiscanner dataset, grouped by scanner device, illustrating scanner-dependent visual variability.
• Figure 4: Low-dimensional visualisation of tile-level embeddings using UMAP for all evaluated feature extractors. For each of the 1920 WSIs in CHIME Multiscanner (384 patients $\times$ 5 scanners $=$ 1920 WSIs), 35 tiles were randomly sampled, projected into a shared two-dimensional embedding space, and coloured according to the scanner device used for acquisition. Each subplot corresponds to a single feature extractor, and marginal plots along the x- and y-axes show the empirical density distributions for each scanner. Distinct clustering or colour separation indicates sensitivity to scanner-induced variability, whereas intermixed distributions suggest greater scanner invariance.
• Figure 5: Low-dimensional visualisation of slide-level embeddings using UMAP for all evaluated feature extractors. Slide-level embeddings were obtained by mean-pooling all tile-level embeddings, for each WSI in the CHIME Multiscanner dataset. Each subplot corresponds to a single feature extractor, with points coloured according to the scanner device used for acquisition. Distinct clustering or colour separation indicates sensitivity to scanner-induced variability, whereas greater overlap suggests increased scanner invariance.