Leveraging Vision-Language Embeddings for Zero-Shot Learning in Histopathology Images

TL;DR

The paper tackles zero-shot learning for histopathology image classification by leveraging vision-language models. It introduces MR-PHE, which fuses multiresolution patch extraction, a hybrid embedding of global and local features, and a prompt-based, domain-informed text representation to align image and text without task-specific fine-tuning. Using CONCH as the shared encoder, the approach applies a similarity-based patch weighting and prompt evaluation/selection to achieve robust image-text alignment, demonstrated across six histopathology datasets with strong results, often surpassing fully supervised methods. Overall, MR-PHE demonstrates that scalable, data-efficient diagnostic guidance is feasible in computational pathology through multimodal embeddings and carefully engineered prompts.

Abstract

Zero-shot learning holds tremendous potential for histopathology image analysis by enabling models to generalize to unseen classes without extensive labeled data. Recent advancements in vision-language models (VLMs) have expanded the capabilities of ZSL, allowing models to perform tasks without task-specific fine-tuning. However, applying VLMs to histopathology presents considerable challenges due to the complexity of histopathological imagery and the nuanced nature of diagnostic tasks. In this paper, we propose a novel framework called Multi-Resolution Prompt-guided Hybrid Embedding (MR-PHE) to address these challenges in zero-shot histopathology image classification. MR-PHE leverages multiresolution patch extraction to mimic the diagnostic workflow of pathologists, capturing both fine-grained cellular details and broader tissue structures critical for accurate diagnosis. We introduce a hybrid embedding strategy that integrates global image embeddings with weighted patch embeddings, effectively combining local and global contextual information. Additionally, we develop a comprehensive prompt generation and selection framework, enriching class descriptions with domain-specific synonyms and clinically relevant features to enhance semantic understanding. A similarity-based patch weighting mechanism assigns attention-like weights to patches based on their relevance to class embeddings, emphasizing diagnostically important regions during classification. Our approach utilizes pretrained VLM, CONCH for ZSL without requiring domain-specific fine-tuning, offering scalability and reducing dependence on large annotated datasets. Experimental results demonstrate that MR-PHE not only significantly improves zero-shot classification performance on histopathology datasets but also often surpasses fully supervised models.

Figures (2)

• Figure 1: An H&E-stained histopathological image of benign breast tissue from the BRACS dataset (resolution: 2472 $\times$ 2370 pixels). The Grad-CAM visualization highlights regions of interest identified by the VLM CONCH model. Warmer colors in the heatmap indicate areas where the model assigns higher attention weights, corresponding to features relevant for benign classification.
• Figure 2: Workflow of our proposed MR-PHE framework. The input image $x$ is divided into multiresolution patches $\{x_1, x_2, \dots, x_N\}$, which are encoded into embeddings $\{e_1, e_2, \dots, e_N\}$ via the image encoder $f$. A hybrid embedding $h$ is created by combining global and weighted patch-level features using attention weights $\{w_1, w_2, \dots, w_N\}$. Simultaneously, class-specific textual prompts $\{T_{c,1}, T_{c,2}, \dots, T_{c,K}\}$ are generated for each class $c$, ranked, and encoded by the text encoder $g$ into prompt embeddings $\{t_{c,1}, t_{c,2}, \dots, t_{c,K}\}$. Text weights $\{v_{c,1}, v_{c,2}, \dots, v_{c,K}\}$ are computed to aggregate these embeddings into the final class embeddings $\tilde{t}_c$. The similarity scores $S_{n,c} = h^\top \tilde{t}_c$ are computed between the hybrid image embedding $h$ and each class embedding $\tilde{t}_c$. These scores $S_{n,c}$ are scaled and converted into class probabilities $P_{n,c}$ using the softmax function. The predicted class label $\hat{y}_n$ is then determined by selecting the class with the highest probability $P_{n,c}$.