Prompt Triage: Structured Optimization Enhances Vision-Language Model Performance on Medical Imaging Benchmarks

TL;DR

This study tackles the underperformance of vision-language models on medical imaging benchmarks by removing reliance on manually engineered prompts and model finetuning. It adapts the DSPy framework to create structured, automated prompt optimization pipelines evaluated across five medical-imaging tasks and ten open-source VLMs, using four optimization techniques (BootstrapFewShotRandomSearch, MIPROv2, GEPA, SIMBA) at inference time. The results show a median relative improvement of 53% over zero-shot baselines, with substantial gains (up to 3,400%) on challenging tasks, and SIMBA emerging as the most effective optimizer across model families. The work demonstrates that prompt optimization offers a scalable, privacy-preserving path to boost medical AI performance without weights modification, and it provides open-source pipelines to enable reproducible, broad adoption in clinical contexts.

Abstract

Vision-language foundation models (VLMs) show promise for diverse imaging tasks but often underperform on medical benchmarks. Prior efforts to improve performance include model finetuning, which requires large domain-specific datasets and significant compute, or manual prompt engineering, which is hard to generalize and often inaccessible to medical institutions seeking to deploy these tools. These challenges motivate interest in approaches that draw on a model's embedded knowledge while abstracting away dependence on human-designed prompts to enable scalable, weight-agnostic performance improvements. To explore this, we adapt the Declarative Self-improving Python (DSPy) framework for structured automated prompt optimization in medical vision-language systems through a comprehensive, formal evaluation. We implement prompting pipelines for five medical imaging tasks across radiology, gastroenterology, and dermatology, evaluating 10 open-source VLMs with four prompt optimization techniques. Optimized pipelines achieved a median relative improvement of 53% over zero-shot prompting baselines, with the largest gains ranging from 300% to 3,400% on tasks where zero-shot performance is low. These results highlight the substantial potential of applying automated prompt optimization to medical AI systems, demonstrating significant gains for vision-based applications requiring accurate clinical image interpretation. By reducing dependence on prompt design to elicit intended outputs, these techniques allow clinicians to focus on patient care and clinical decision-making. Furthermore, our experiments offer scalability and preserve data privacy, demonstrating performance improvement on open-source VLMs. We publicly release our evaluation pipelines to support reproducible research on specialized medical tasks, available at https://github.com/DaneshjouLab/prompt-triage-lab.

Figures (1)

• Figure 1: Overview of DSPy Prompt Triage for medical vision-language models. We benchmark five medical imaging tasks (VQA-RAD, GastroVision, CheXpert, DDI Skin Tone, and DDI Diagnosis) across open-source VLM families (Qwen, Gemma/MedGemma, Llama). Baselines use handwritten prompts, while DSPy Prompt Triage applies various prompt optimization techniques (BootstrapFewShotRandom Search, MIPROv2, GEPA, SIMBA), yielding performance gains over the baselines.