Benchmarking Pathology Feature Extractors for Whole Slide Image Classification

TL;DR

This study constitutes the most comprehensive evaluation of publicly available pathology feature extractors to date, involving more than 10,000 training runs across 14 feature extractors, 9 tasks, 5 datasets, 3 downstream architectures, 2 levels of magnification, and various preprocessing setups.

Abstract

Weakly supervised whole slide image classification is a key task in computational pathology, which involves predicting a slide-level label from a set of image patches constituting the slide. Constructing models to solve this task involves multiple design choices, often made without robust empirical or conclusive theoretical justification. To address this, we conduct a comprehensive benchmarking of feature extractors to answer three critical questions: 1) Is stain normalisation still a necessary preprocessing step? 2) Which feature extractors are best for downstream slide-level classification? 3) How does magnification affect downstream performance? Our study constitutes the most comprehensive evaluation of publicly available pathology feature extractors to date, involving more than 10,000 training runs across 14 feature extractors, 9 tasks, 5 datasets, 3 downstream architectures, 2 levels of magnification, and various preprocessing setups. Our findings challenge existing assumptions: 1) We observe empirically, and by analysing the latent space, that skipping stain normalisation and image augmentations does not degrade performance, while significantly reducing memory and computational demands. 2) We develop a novel evaluation metric to compare relative downstream performance, and show that the choice of feature extractor is the most consequential factor for downstream performance. 3) We find that lower-magnification slides are sufficient for accurate slide-level classification. Contrary to previous patch-level benchmarking studies, our approach emphasises clinical relevance by focusing on slide-level biomarker prediction tasks in a weakly supervised setting with external validation cohorts. Our findings stand to streamline digital pathology workflows by minimising preprocessing needs and informing the selection of feature extractors.

Figures (17)

• Figure 1: Common setup for weakly supervised learning on . In the preprocessing stage, the input image is split into patches that undergo independent image augmentations $a$ before feature extraction. The feature aggregator and classifier are trained jointly as a single neural network $g_\theta \circ h_\theta$, which, given feature vectors as inputs, predicts the output $y$. For stain normalisation (shown here), the same $a(\cdot)$ is applied every time, though in general, the augmentation function may vary between patches and epochs.
• Figure 2: Latent space visualisations (t-SNE vandermaaten2008visualizing) of features extracted with Lunit-DINO (left) vs. ImageNet baseline (right). Colours represent tissue classes kather2018100000. Both feature extractors use the same architecture (ViT-S), but the left was trained on pathology images using . Each dot represents a feature vector in latent space extracted from an unaltered image patch, and we draw a line from that dot to the corresponding stain-normalised version.
• Figure 3: Boxplot of cosine distances between patch embeddings and their stain-normalised versions, as well as between embeddings of randomly chosen patches of the same or of differing tissue types. Feature extractors are grouped by architecture (ImageNet baselines are bold). Whiskers represent 95% of the distances.
• Figure 4: Boxplot of embedding displacement induced by image augmentations for Lunit-DINO and ViT-S. Dashed lines represent the average distance between randomly selected patches (without augmentation) indicating how 'dispersed' the latent spaces are.
• Figure 5: Visualisations of latent space transformations caused by rotation-based augmentations (rows) in Lunit-DINO (left) and ViT-S (right). Colours and lines are as explained in \ref{['fig:latent_macenko']}. Top row: 90° rotation. Middle: rotating by a random angle. Bottom (ablation): each line represents the transformation from (a) the embedding of the $1.5\times$ zoomed version of a patch to (b) the embedding obtained by randomly rotating before the $1.5\times$ zoom.