Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design

Vasudev Sharma, Ahmed Alagha, Abdelhakim Khellaf, Vincent Quoc-Huy Trinh, Mahdi S. Hosseini

TL;DR

Vision-language models for diagnostic pathology show sensitivity to data scale, task formulation, and prompts. The study systematically evaluates Quilt-Net, Quilt-LLAVA, and CONCH on 3,507 digestive giga-pixel WSIs using a structured prompt-engineering framework across four design dimensions, revealing that anatomical precision and domain-specific alignment dominate performance while larger models do not guarantee superiority. CONCH achieves the best accuracy and localization of malignant regions, outperforming larger counterparts when guided by precise anatomical prompts. These findings offer actionable guidelines for prompt design and attention-based validation to enhance AI-assisted histopathology diagnostics in clinical settings.

Abstract

Vision-language models (VLMs) have gained significant attention in computational pathology due to their multimodal learning capabilities that enhance big-data analytics of giga-pixel whole slide image (WSI). However, their sensitivity to large-scale clinical data, task formulations, and prompt design remains an open question, particularly in terms of diagnostic accuracy. In this paper, we present a systematic investigation and analysis of three state of the art VLMs for histopathology, namely Quilt-Net, Quilt-LLAVA, and CONCH, on an in-house digestive pathology dataset comprising 3,507 WSIs, each in giga-pixel form, across distinct tissue types. Through a structured ablative study on cancer invasiveness and dysplasia status, we develop a comprehensive prompt engineering framework that systematically varies domain specificity, anatomical precision, instructional framing, and output constraints. Our findings demonstrate that prompt engineering significantly impacts model performance, with the CONCH model achieving the highest accuracy when provided with precise anatomical references. Additionally, we identify the critical importance of anatomical context in histopathological image analysis, as performance consistently degraded when reducing anatomical precision. We also show that model complexity alone does not guarantee superior performance, as effective domain alignment and domain-specific training are critical. These results establish foundational guidelines for prompt engineering in computational pathology and highlight the potential of VLMs to enhance diagnostic accuracy when properly instructed with domain-appropriate prompts.

Investigating Zero-Shot Diagnostic Pathology in Vision-Language Models with Efficient Prompt Design

TL;DR

Vision-language models for diagnostic pathology show sensitivity to data scale, task formulation, and prompts. The study systematically evaluates Quilt-Net, Quilt-LLAVA, and CONCH on 3,507 digestive giga-pixel WSIs using a structured prompt-engineering framework across four design dimensions, revealing that anatomical precision and domain-specific alignment dominate performance while larger models do not guarantee superiority. CONCH achieves the best accuracy and localization of malignant regions, outperforming larger counterparts when guided by precise anatomical prompts. These findings offer actionable guidelines for prompt design and attention-based validation to enhance AI-assisted histopathology diagnostics in clinical settings.

Abstract

Vision-language models (VLMs) have gained significant attention in computational pathology due to their multimodal learning capabilities that enhance big-data analytics of giga-pixel whole slide image (WSI). However, their sensitivity to large-scale clinical data, task formulations, and prompt design remains an open question, particularly in terms of diagnostic accuracy. In this paper, we present a systematic investigation and analysis of three state of the art VLMs for histopathology, namely Quilt-Net, Quilt-LLAVA, and CONCH, on an in-house digestive pathology dataset comprising 3,507 WSIs, each in giga-pixel form, across distinct tissue types. Through a structured ablative study on cancer invasiveness and dysplasia status, we develop a comprehensive prompt engineering framework that systematically varies domain specificity, anatomical precision, instructional framing, and output constraints. Our findings demonstrate that prompt engineering significantly impacts model performance, with the CONCH model achieving the highest accuracy when provided with precise anatomical references. Additionally, we identify the critical importance of anatomical context in histopathological image analysis, as performance consistently degraded when reducing anatomical precision. We also show that model complexity alone does not guarantee superior performance, as effective domain alignment and domain-specific training are critical. These results establish foundational guidelines for prompt engineering in computational pathology and highlight the potential of VLMs to enhance diagnostic accuracy when properly instructed with domain-appropriate prompts.
Paper Structure (8 sections, 7 figures, 4 tables)

This paper contains 8 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Benchmarking on Big-Data Cohort of Digestive Pathology
  4. Evaluation Methodology
  5. Results
  6. Ablative study
  7. Attention Maps Analysis
  8. Conclusion

Figures (7)

  • Figure 1: High level overview of the inference process for the three VLMs.
  • Figure 2: Sample images from the in-house dataset
  • Figure 3: ROC curves comparing the performance of three vision-language models: (a) Quilt-Net, (b) Quilt-LLAVA, and (c) CONCH, across different prompts for invasive cancer classification. CONCH demonstrates the highest robustness and performance consistency, while Quilt-Net and Quilt-LLAVA exhibit significant sensitivity to prompt design.
  • Figure 4: Performance comparison of VLM models in terms of ROC AUC. CONCH consistently outperforms the other models despite having fewer parameters than Quilt-LLAVA, underscoring the importance of domain-specific training and prompt alignment over model scale.
  • Figure 5: Performance comparison of VLM models and prompts in terms of ROC AUC for dysplasia status.
  • ...and 2 more figures