DualPrompt-MedCap: A Dual-Prompt Enhanced Approach for Medical Image Captioning

TL;DR

This work tackles the core challenges of medical image captioning by targeting unreliable modality recognition and the need for clinically meaningful descriptions under limited labeled data. It introduces DualPrompt-MedCap, a dual-prompt framework that fuses a modality-aware prompt learned through semi-supervised modality learning with a question-guided clinical focus prompt, all integrated into a BLIP3-based captioning pipeline. A ground-truth-independent evaluation framework combines image-text and question-text relevance with radiology-quality metrics to assess clinical utility without direct references. Empirical results on RAD and SLAKE show substantial gains in modality accuracy and in generating question-aligned, clinically accurate captions, indicating strong potential to aid medical professionals and automate medical annotations. The approach offers a scalable path for enhancing diagnostic reporting in resource-limited settings and for downstream vision-language tasks in healthcare.

Abstract

Medical image captioning via vision-language models has shown promising potential for clinical diagnosis assistance. However, generating contextually relevant descriptions with accurate modality recognition remains challenging. We present DualPrompt-MedCap, a novel dual-prompt enhancement framework that augments Large Vision-Language Models (LVLMs) through two specialized components: (1) a modality-aware prompt derived from a semi-supervised classification model pretrained on medical question-answer pairs, and (2) a question-guided prompt leveraging biomedical language model embeddings. To address the lack of captioning ground truth, we also propose an evaluation framework that jointly considers spatial-semantic relevance and medical narrative quality. Experiments on multiple medical datasets demonstrate that DualPrompt-MedCap outperforms the baseline BLIP-3 by achieving a 22% improvement in modality recognition accuracy while generating more comprehensive and question-aligned descriptions. Our method enables the generation of clinically accurate reports that can serve as medical experts' prior knowledge and automatic annotations for downstream vision-language tasks.

Figures (4)

• Figure 1: Overview of the proposed DualPrompt-MedCap framework.
• Figure 2: Semi-supervised modality learning module.
• Figure 3: DualPrompt-MedCap caption analysis showing relevance and quality.
• Figure 4: DualPrompt-MedCap with other models in medical image captioning.