Unlocking adaptive digital pathology through dynamic feature learning

TL;DR

This work introduces PathFiT, a dynamic feature learning method that can be effortlessly plugged into various pathology foundation models to unlock their adaptability, and demonstrates state-of-the-art performance on 34 out of 35 tasks.

Abstract

Foundation models have revolutionized the paradigm of digital pathology, as they leverage general-purpose features to emulate real-world pathological practices, enabling the quantitative analysis of critical histological patterns and the dissection of cancer-specific signals. However, these static general features constrain the flexibility and pathological relevance in the ever-evolving needs of clinical applications, hindering the broad use of the current models. Here we introduce PathFiT, a dynamic feature learning method that can be effortlessly plugged into various pathology foundation models to unlock their adaptability. Meanwhile, PathFiT performs seamless implementation across diverse pathology applications regardless of downstream specificity. To validate PathFiT, we construct a digital pathology benchmark with over 20 terabytes of Internet and real-world data comprising 28 H\&E-stained tasks and 7 specialized imaging tasks including Masson's Trichrome staining and immunofluorescence images. By applying PathFiT to the representative pathology foundation models, we demonstrate state-of-the-art performance on 34 out of 35 tasks, with significant improvements on 23 tasks and outperforming by 10.20% on specialized imaging tasks. The superior performance and versatility of PathFiT open up new avenues in computational pathology.

Figures (16)

• Figure : Figure 1: Overview of PathFiT.a. The typical paradigm in computational pathology is to use a series of tissue-contain patches as basic units, convert them into sequential image tokens, and feed them into transformer-based foundation models for forward modeling. b. The difference in downstream adaptation workflow between general feature learning and dynamic feature-based PathFiT. In the conventional process, only the parameters of the classifier layer are updated, while the weights within the foundation model remain unchanged. In contrast, PathFiT insets lightweight, trainable modules into the pretrained foundation model, enabling backpropagation to not only update the classifier but also dynamically adjust image features through the additional parameters to better adapt to downstream tasks. (Next page.)
• Figure : (Previous page.) Figure 1: Overview of PathFiT.c. PathFiT adds extra parameters in parallel to the linear layers within the self-attention of each transformer block. This design allows for dynamic adjustment of feature outputs while preserving the original model weights. d. PathFiT improves the performance of the visual-language foundation model CONCH on all tasks as well as fine-grained tasks, rare disease tasks, and specialized imaging tasks. e. PathFiT improves the performance of the visual foundation model UNI on all tasks as well as fine-grained tasks, rare disease tasks, and specialized imaging tasks.
• Figure : Figure 2: ROI-level supervised classification.a. By enabling PathFiT, foundation models pretrained on H&E-stained image patches are adapted to ROI-level tasks at different resolutions. b. By enabling PathFiT, CONCH increased macro AUC from 96.29% to 98.15%, and UNI increased from 97.03% to 98.47%. c. By enabling PathFiT, CONCH decreased balanced error from 14.50% to 8.92%, and UNI decreased from 12.02% to 7.54%. d,e. Balanced accuracy comparison of CONCH and UNI across all ROI-level tasks between disabling and enabling PathFiT. f. Text prompt few-shot learning comparison between disabling and enabling PathFiT on CRC and ESCA tissue classification tasks. g. Comparison across different ROI resolutions between disabling and enabling PathFiT on BRCA fine-grained subtyping and OS tumor tissue classification. h. Visualization comparison of image embeddings between disabling and enabling PathFiT on the BRCA conventional subtyping task. i. Multi-head self-attention heatmap comparison with disabling and enabling PathFiT.
• Figure : Figure 3: Slide-level supervised classification. Caption on next page.
• Figure : (Previous page.) Figure 3: Slide-level supervised classification.a. By enabling PathFiT, foundation models pretrained on H&E-stained image patches are adapted to resection and biopsy WSI tasks. b. By enabling PathFiT, CONCH increased macro AUC from 90.32% to 90.58%, and UNI increased from 88.53% to 90.26%. c. By enabling PathFiT, CONCH decreased balanced error from 31.68% to 29.54%, and UNI decreased from 33.60% to 30.29%. d,e. Balanced accuracy comparison of CONCH and UNI across all resection WSI tasks between disabling and enabling PathFiT. f. An average AUC of 97.39%, 90.02%, and 91.22% was achieved for biopsy PRAD screening, PRAD grading, and cervical inflammatory tissue classification tasks with PathFiT enabled. g. Few-shot learning comparison between disabling and enabling PathFiT on TCGA OncoTree classification. h. Visualization comparison of image embeddings between disabling and enabling PathFiT on PRAD grading tasks. i. Attention weight heatmaps of the MIL aggregator between disabling and enabling PathFiT.