Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Vision-Language Models for Histopathology Classification: Predict all at once

Maxime Zanella, Fereshteh Shakeri, Yunshi Huang, Houda Bahig, Ismail Ben Ayed

TL;DR

This work addresses automating histopathology tissue classification over whole-slide images by moving beyond inductive patchwise prediction to a transductive, text‑assisted inference paradigm. The authors introduce Histo-TransCLIP, which operates entirely in the embedding space and combines a Gaussian mixture model, text-based pseudo-labels from class prompts, and a Laplacian regularization over patch affinities, with a KL penalty to align predictions with the zero-shot priors. Across four histology datasets and five vision‑language models, the method yields substantial accuracy gains over standard zero-shot inference while remaining computationally efficient for large patch counts (e.g., processing 10^5 patches in seconds). The approach requires no additional labels and does not access the model parameters, making it scalable and suitable for clinical-scale histopathology; future work includes few-shot extensions and prompt-quality analyses to further improve safety and performance.

Abstract

The development of vision-language models (VLMs) for histo-pathology has shown promising new usages and zero-shot performances. However, current approaches, which decompose large slides into smaller patches, focus solely on inductive classification, i.e., prediction for each patch is made independently of the other patches in the target test data. We extend the capability of these large models by introducing a transductive approach. By using text-based predictions and affinity relationships among patches, our approach leverages the strong zero-shot capabilities of these new VLMs without any additional labels. Our experiments cover four histopathology datasets and five different VLMs. Operating solely in the embedding space (i.e., in a black-box setting), our approach is highly efficient, processing $10^5$ patches in just a few seconds, and shows significant accuracy improvements over inductive zero-shot classification. Code available at https://github.com/FereshteShakeri/Histo-TransCLIP.

Boosting Vision-Language Models for Histopathology Classification: Predict all at once

TL;DR

This work addresses automating histopathology tissue classification over whole-slide images by moving beyond inductive patchwise prediction to a transductive, text‑assisted inference paradigm. The authors introduce Histo-TransCLIP, which operates entirely in the embedding space and combines a Gaussian mixture model, text-based pseudo-labels from class prompts, and a Laplacian regularization over patch affinities, with a KL penalty to align predictions with the zero-shot priors. Across four histology datasets and five vision‑language models, the method yields substantial accuracy gains over standard zero-shot inference while remaining computationally efficient for large patch counts (e.g., processing 10^5 patches in seconds). The approach requires no additional labels and does not access the model parameters, making it scalable and suitable for clinical-scale histopathology; future work includes few-shot extensions and prompt-quality analyses to further improve safety and performance.

Abstract

The development of vision-language models (VLMs) for histo-pathology has shown promising new usages and zero-shot performances. However, current approaches, which decompose large slides into smaller patches, focus solely on inductive classification, i.e., prediction for each patch is made independently of the other patches in the target test data. We extend the capability of these large models by introducing a transductive approach. By using text-based predictions and affinity relationships among patches, our approach leverages the strong zero-shot capabilities of these new VLMs without any additional labels. Our experiments cover four histopathology datasets and five different VLMs. Operating solely in the embedding space (i.e., in a black-box setting), our approach is highly efficient, processing patches in just a few seconds, and shows significant accuracy improvements over inductive zero-shot classification. Code available at https://github.com/FereshteShakeri/Histo-TransCLIP.
Paper Structure (19 sections, 7 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 19 sections, 7 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. VLMs for histology.
  5. Transductive learning.
  6. Method
  7. Gaussian modelization
  8. Text-based predictions.
  9. Laplacian regularization.
  10. Objective function.
  11. Procedure.
  12. Experiments
  13. Datasets.
  14. Results.
  15. Computational workload.
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Illustration depicting histopathology classification in the inductive setting (a), the commonly-used few-shot transductive setting (b), and the zero-shot transductive setting enabled by VLMs (c).