Rethinking Pre-Trained Feature Extractor Selection in Multiple Instance Learning for Whole Slide Image Classification

TL;DR

This paper tackles the problem of selecting pre-trained feature extractors for embedding-based MIL in gigapixel WSI classification by proposing a three-dimensional analysis over pre-training dataset size/variety, backbone design, and pre-training method. Using two public datasets, TCGA-NSCLC and Camelyon16, and four MIL models, the study demonstrates that the choice of self-supervised pre-training method and Transformer-based backbones with larger pre-training datasets yield the most consistent performance gains, even more than in-domain pre-training alone. The findings explain why modern pathology foundation models benefit from diverse, large-scale pre-training and advanced SSL techniques, and offer concrete guidance for building more effective MIL pipelines for histopathology. Overall, the work provides practical recommendations for feature-extractor selection and helps justify the performance advantages of current pathology foundation models, with implications for future model design and data curation in pathology.

Abstract

Multiple instance learning (MIL) has become a preferred method for gigapixel whole slide image (WSI) classification without requiring patch-level annotations. Current MIL research primarily relies on embedding-based approaches, which extract patch features using a pre-trained feature extractor and aggregate them for slide-level prediction. Despite the critical role of feature extraction, there is limited guidance on selecting optimal feature extractors to maximize WSI performance. This study addresses this gap by systematically evaluating MIL feature extractors across three dimensions: pre-training dataset, backbone model, and pre-training method. Extensive experiments were conducted on two public WSI datasets (TCGA-NSCLC and Camelyon16) using four state-of-the-art (SOTA) MIL models. Our findings reveal that: 1) selecting a robust self-supervised learning (SSL) method has a greater impact on performance than relying solely on an in-domain pre-training dataset; 2) prioritizing Transformer-based backbones with deeper architectures over CNN-based models; and 3) using larger, more diverse pre-training datasets significantly enhances classification outcomes. We hope that these insights can provide practical guidance for optimizing WSI classification and explain the reasons behind the performance advantages of the current SOTA pathology foundation models. Furthermore, this work may inform the development of more effective pathology foundation models. Our code is publicly available at https://github.com/bryanwong17/MIL-Feature-Extractor-Selection

Figures (2)

• Figure 1: Analytical setup overview for optimizing MIL pre-trained feature extractors.
• Figure 2: The effect of using a larger and more varied pre-training dataset (ImageNet-1K ImageNet-1K vs. ImageNet-21K ImageNet-21K) on CNN and Transformer backbones of feature extractors.