Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PathoTune: Adapting Visual Foundation Model to Pathological Specialists

Jiaxuan Lu, Fang Yan, Xiaofan Zhang, Yue Gao, Shaoting Zhang

TL;DR

PathoTune addresses the challenge of adapting broad foundation models to pathology tasks by identifying two domain gaps, $FTG$ and $TIG$, and introducing a multi-modal prompt design (TVP, TTP, IVP) plus a Visual Refine Module to encode instance-specific features. It demonstrates that prompting-based adaptation of a visual or pathological foundation model can outperform linear probing and approach full finetuning with a small parameter budget. The method yields superior performance on patch-level and WSI-level pathology tasks across diverse datasets and stains, often surpassing state-of-the-art PEFT methods. This work enables efficient reuse of existing foundation models for pathology, reducing task-specific retraining while maintaining high accuracy, with code available.

Abstract

As natural image understanding moves towards the pretrain-finetune era, research in pathology imaging is concurrently evolving. Despite the predominant focus on pretraining pathological foundation models, how to adapt foundation models to downstream tasks is little explored. For downstream adaptation, we propose the existence of two domain gaps, i.e., the Foundation-Task Gap and the Task-Instance Gap. To mitigate these gaps, we introduce PathoTune, a framework designed to efficiently adapt pathological or even visual foundation models to pathology-specific tasks via multi-modal prompt tuning. The proposed framework leverages Task-specific Visual Prompts and Task-specific Textual Prompts to identify task-relevant features, along with Instance-specific Visual Prompts for encoding single pathological image features. Results across multiple datasets at both patch-level and WSI-level demonstrate its superior performance over single-modality prompt tuning approaches. Significantly, PathoTune facilitates the direct adaptation of natural visual foundation models to pathological tasks, drastically outperforming pathological foundation models with simple linear probing. The code is available at https://github.com/openmedlab/PathoDuet.

PathoTune: Adapting Visual Foundation Model to Pathological Specialists

TL;DR

PathoTune addresses the challenge of adapting broad foundation models to pathology tasks by identifying two domain gaps, and , and introducing a multi-modal prompt design (TVP, TTP, IVP) plus a Visual Refine Module to encode instance-specific features. It demonstrates that prompting-based adaptation of a visual or pathological foundation model can outperform linear probing and approach full finetuning with a small parameter budget. The method yields superior performance on patch-level and WSI-level pathology tasks across diverse datasets and stains, often surpassing state-of-the-art PEFT methods. This work enables efficient reuse of existing foundation models for pathology, reducing task-specific retraining while maintaining high accuracy, with code available.

Abstract

As natural image understanding moves towards the pretrain-finetune era, research in pathology imaging is concurrently evolving. Despite the predominant focus on pretraining pathological foundation models, how to adapt foundation models to downstream tasks is little explored. For downstream adaptation, we propose the existence of two domain gaps, i.e., the Foundation-Task Gap and the Task-Instance Gap. To mitigate these gaps, we introduce PathoTune, a framework designed to efficiently adapt pathological or even visual foundation models to pathology-specific tasks via multi-modal prompt tuning. The proposed framework leverages Task-specific Visual Prompts and Task-specific Textual Prompts to identify task-relevant features, along with Instance-specific Visual Prompts for encoding single pathological image features. Results across multiple datasets at both patch-level and WSI-level demonstrate its superior performance over single-modality prompt tuning approaches. Significantly, PathoTune facilitates the direct adaptation of natural visual foundation models to pathological tasks, drastically outperforming pathological foundation models with simple linear probing. The code is available at https://github.com/openmedlab/PathoDuet.
Paper Structure (10 sections, 6 equations, 4 figures, 1 table)

This paper contains 10 sections, 6 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Multi-Modal Prompt Design
  6. Overall Procedure
  7. Experiments and Results
  8. Conclusion
  9. Acknowledgments.
  10. Disclosure of Interests.

Figures (4)

  • Figure 1: Compared to traditional paradigms of training separate models for each task or training a pathological foundation model, PathoTune directly adapts a visual or pathological foundation model to downstream tasks using multi-modal prompts.
  • Figure 2: Overview of the proposed PathoTune. (A) The input and output of PathoTune for both patch-level and WSI-level tasks. (B) Detailed architecture of PathoTune, encompassing the Task-specific Visual Prompts (TVP), Task-specific Textual Prompts (TTP) and Instance-specific Visual Prompts (IVP).
  • Figure 3: Comparisons of the PathoTune with other SOTA methods of PEFT.
  • Figure 4: Comparisons of the PathoTune with different prompt combination.