PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

TL;DR

PathoDuet addresses the data-hungry bottleneck of histopathology foundation models by introducing a pretext-token SSL framework with two targeted tasks. The cross-scale positioning task enhances H&E understanding by linking local patches to global context, while cross-stain transferring leverages H&E structure to interpret IHC slides via AdaIN-based feature transfer. Through two-stage pretraining and extensive downstream evaluation on both H&E and IHC tasks, PathoDuet demonstrates improved performance and data efficiency relative to conventional ImageNet and domain-specific baselines, and shows competitive results against giant pathology models with far less data. The work highlights the practical impact of tailored SSL design for pathology and provides a reusable open-source pipeline for advancing histopathology foundation models.

Abstract

Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. To validate the efficacy of our models, we evaluate the performance over a wide variety of downstream tasks, including patch-level colorectal cancer subtyping and whole slide image (WSI)-level classification in H&E field, together with expression level prediction of IHC marker, tumor identification and slide-level qualitative analysis in IHC field. The experimental results show the superiority of our models over most tasks and the efficacy of proposed pretext tasks. The codes and models are available at https://github.com/openmedlab/PathoDuet.

Figures (5)

• Figure 1: An overview of PathoDuet. Left: two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to develop H&E and IHC models. Right: a series of downstream tasks, covering both H&E and IHC ones, are used to evaluate models' performance in application.
• Figure 2: An overall performance visualization. Each task is named as training dataset, (special settings,) evaluating metric. H&E tasks are colored purple, and IHC ones are yellow.
• Figure 3: Detailed networks of two pretext tasks. The flow of pretext token is represented by the black arrows, $\epsilon$ is the placeholder, and the task raisers (positioner and transferer) are presented in the white blocks. (a) Three-branch cross-scale positioning network. (b) Two-branch cross-stain transferring network and the transferer module.
• Figure 4: Data requirement study on NCT-CRC-HE dataset. PathoDuet is compared with ImageSup and the performance with the full dataset as an upper bound is represented by the dotted line with the same color.
• Figure 5: Data requirement study on different datasets using PathoDuet.