Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models

Wei Peng, Kang Liu, Jianchen Hu, Meng Zhang

TL;DR

Biomed-DPT addresses the domain gap in biomedical vision-language tasks by introducing a knowledge-enhanced dual-modality prompt-tuning framework. It combines fixed clinical text prompts with GPT-4–generated domain prompts and employs a zero-vector soft visual prompt to reweight image attention, all optimized with cross-entropy, $L_1$, and KL-divergence losses to distill external clinical knowledge into the model. On 11 biomedical datasets, Biomed-DPT achieves average accuracy of $66.14\%$ with base-class accuracy of $78.06\%$ and novel-class accuracy of $75.97\%$, outperforming CoOp by notable margins. The approach demonstrates strong few-shot and base-to-novel generalization and provides improved interpretability of lesion localization, illustrating its practical potential for robust biomedical image analysis.

Abstract

Prompt learning is one of the most effective paradigms for adapting pre-trained vision-language models (VLMs) to the biomedical image classification tasks in few shot scenarios. However, most of the current prompt learning methods only used the text prompts and ignored the particular structures (such as the complex anatomical structures and subtle pathological features) in the biomedical images. In this work, we propose Biomed-DPT, a knowledge-enhanced dual modality prompt tuning technique. In designing the text prompt, Biomed-DPT constructs a dual prompt including the template-driven clinical prompts and the large language model (LLM)-driven domain-adapted prompts, then extracts the clinical knowledge from the domain-adapted prompts through the knowledge distillation technique. In designing the vision prompt, Biomed-DPT introduces the zero vector as a soft prompt to leverage attention re-weighting so that the focus on non-diagnostic regions and the recognition of non-critical pathological features are avoided. Biomed-DPT achieves an average classification accuracy of 66.14\% across 11 biomedical image datasets covering 9 modalities and 10 organs, with performance reaching 78.06\% in base classes and 75.97\% in novel classes, surpassing the Context Optimization (CoOp) method by 6.20\%, 3.78\%, and 8.04\%, respectively. Our code are available at \underline{https://github.com/Kanyooo/Biomed-DPT}.

Biomed-DPT: Dual Modality Prompt Tuning for Biomedical Vision-Language Models

TL;DR

Biomed-DPT addresses the domain gap in biomedical vision-language tasks by introducing a knowledge-enhanced dual-modality prompt-tuning framework. It combines fixed clinical text prompts with GPT-4–generated domain prompts and employs a zero-vector soft visual prompt to reweight image attention, all optimized with cross-entropy, , and KL-divergence losses to distill external clinical knowledge into the model. On 11 biomedical datasets, Biomed-DPT achieves average accuracy of with base-class accuracy of and novel-class accuracy of , outperforming CoOp by notable margins. The approach demonstrates strong few-shot and base-to-novel generalization and provides improved interpretability of lesion localization, illustrating its practical potential for robust biomedical image analysis.

Abstract

Prompt learning is one of the most effective paradigms for adapting pre-trained vision-language models (VLMs) to the biomedical image classification tasks in few shot scenarios. However, most of the current prompt learning methods only used the text prompts and ignored the particular structures (such as the complex anatomical structures and subtle pathological features) in the biomedical images. In this work, we propose Biomed-DPT, a knowledge-enhanced dual modality prompt tuning technique. In designing the text prompt, Biomed-DPT constructs a dual prompt including the template-driven clinical prompts and the large language model (LLM)-driven domain-adapted prompts, then extracts the clinical knowledge from the domain-adapted prompts through the knowledge distillation technique. In designing the vision prompt, Biomed-DPT introduces the zero vector as a soft prompt to leverage attention re-weighting so that the focus on non-diagnostic regions and the recognition of non-critical pathological features are avoided. Biomed-DPT achieves an average classification accuracy of 66.14\% across 11 biomedical image datasets covering 9 modalities and 10 organs, with performance reaching 78.06\% in base classes and 75.97\% in novel classes, surpassing the Context Optimization (CoOp) method by 6.20\%, 3.78\%, and 8.04\%, respectively. Our code are available at \underline{https://github.com/Kanyooo/Biomed-DPT}.
Paper Structure (23 sections, 10 equations, 8 figures, 11 tables)

This paper contains 23 sections, 10 equations, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Few-shot Adaptation of VLMs
  5. Methodology
  6. Clinical knowledge enhancement
  7. Learnable Token Text Prompt
  8. Zero-vector As A Soft Visual Prompt
  9. Experiments and Results
  10. Experimental Setup
  11. Prompt Template Evaluation
  12. Few-shot Evaluation
  13. Base-to-Novel Generalization
  14. Ablation experiments
  15. Effect of Different Components
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Three categories of fine-tuning methods in VLMs. The dashed lines indicate potential interaction relationships between prompts.
  • Figure 1: Average classification accuracy (%) of various fewshot adaptation methods across different numbers of training shots per class.
  • Figure 2: Effect of CLIP pre-tained on natural datasets vs BiomedCLIP pre-tained on biomedical datasets on visual saliency maps. Red lines indicate foreground regions, yellow lines represent background regions associated with object.
  • Figure 2: Comparison with differnet context on the 11 datasets. "front" or "mid" or "end" means putting the class token in the front or middle or end. TSC means task-specific context. BSC means biomedical-specific context. CSC means class-specific context.
  • Figure 3: Comparison of classification accuracies (%) of different initialization methods. [CLS] denotes [CLASS].
  • ...and 3 more figures