MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification

TL;DR

This work introduces MGPATH, a parameter-efficient vision-language framework for few-shot whole-slide pathology classification that bridges Prov-GigaPath visual representations with PLIP text embeddings via lightweight adaptors. It implements multi-granular prompt learning with prompts at both patch and region levels, guided by dual-scale descriptive prompts generated from an LLM, and couples them through an optimal-transport distance to achieve robust visual-text alignment. The approach demonstrates consistent improvements over MIL and prior VLMs across NSCLC, RCC, and BRCA datasets, including strong performance in zero-shot and few-shot settings and across multiple backbones such as CLIP, PLIP, and ViT-based architectures. These results underscore the value of combining large-scale pathology pretraining with hierarchical prompts and OT-based alignment to enhance generalization in data-sparse pathology tasks, with potential for broader adoption in research and clinical-context AI tools.

Abstract

Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels, hindering model generalization. This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification. We first extend the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology image tiles, into a vision-language model by adding adaptors and aligning it with medical text encoders via contrastive learning on 923K image-text pairs. The model is then used to extract visual features and text embeddings from few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike prior methods that combine prompts with frozen features using prefix embeddings or self-attention, we propose multi-granular attention that compares interactions between learnable prompts with individual image patches and groups of them. This approach improves the model's ability to capture both fine-grained details and broader context, enhancing its recognition of complex patterns across sub-regions. To further improve accuracy, we leverage (unbalanced) optimal transport-based visual-text distance to secure model robustness by mitigating perturbations that might occur during the data augmentation process. Empirical experiments on lung, kidney, and breast pathology modalities validate the effectiveness of our approach; thereby, we surpass several of the latest competitors and consistently improve performance across diverse architectures, including CLIP, PLIP, and Prov-GigaPath integrated PLIP.

Figures (5)

• Figure 1: Unlike previous methods that add prompts at prefix positions or patch-level attention - disrupting structural correlations - our MGPath framework integrates prompts at both regional and individual patch levels (multi-granular attention). The tokens $t+1$, $t+2$, and $t+3$ represent the spatially adjacent patches within a larger tissue region (highlighted by the larger orange box), which together contribute to the model's diagnostic decision. In contrast, $t+9$ represents a more distant or isolated patch froma different region.
• Figure 2: The pipeline of the proposed MGPATH. Low- and high-resolution image patches are processed with large language models to generate visual contextual descriptions (Section \ref{['sec:text_prompt']}). Visual prompts are integrated with frozen features through multi-granular attention at both patch and group-of-patch levels \ref{['sec:visual_prompt']}. The final output is obtained by aligning visual and text embeddings using optimal transport (Section \ref{['sec:ot_distance']}).
• Figure 3: LLM template prompt.
• Figure 4: Comparison of methods on TCGA-BRCA with 16-shot settings.
• Figure 5: F1 scores of MGPATH(PLIP-G) under 2-, 4-, 8-, and 16-shots training on TCGA-BRCA.