BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models

TL;DR

BiomedXPro presents an evolutionary prompting framework that uses a biomedical LLM as both knowledge extractor and optimizer to generate a diverse ensemble of interpretable, language-based prompt pairs for biomedical image diagnosis. By optimizing a population of prompt pairs through fitness-guided evolution and LLM-driven mutations, it achieves strong few-shot generalization and aligns discovered prompts with clinically meaningful features. The approach yields transparent decision rationales via interpretable prompts and weighted voting, addressing critical barriers to clinical adoption of vision-language models. Across Derm7pt, WBCAtt, and Camelyon17-WILDS, BiomedXPro outperforms state-of-the-art prompt-tuning methods in data-scarce settings, while maintaining competitive performance with full data and providing a verifiable interpretability framework for clinical trustworthiness.

Abstract

The clinical adoption of biomedical vision-language models is hindered by prompt optimization techniques that produce either uninterpretable latent vectors or single textual prompts. This lack of transparency and failure to capture the multi-faceted nature of clinical diagnosis, which relies on integrating diverse observations, limits their trustworthiness in high-stakes settings. To address this, we introduce BiomedXPro, an evolutionary framework that leverages a large language model as both a biomedical knowledge extractor and an adaptive optimizer to automatically generate a diverse ensemble of interpretable, natural-language prompt pairs for disease diagnosis. Experiments on multiple biomedical benchmarks show that BiomedXPro consistently outperforms state-of-the-art prompt-tuning methods, particularly in data-scarce few-shot settings. Furthermore, our analysis demonstrates a strong semantic alignment between the discovered prompts and statistically significant clinical features, grounding the model's performance in verifiable concepts. By producing a diverse ensemble of interpretable prompts, BiomedXPro provides a verifiable basis for model predictions, representing a critical step toward the development of more trustworthy and clinically-aligned AI systems.

Figures (5)

• Figure 1: An overview of the BiomedXPro evolutionary framework. At each iteration, a LLM generates new prompt pairs ($P$) guided by a meta-prompt ($Q_t$) containing a subset of previously successful candidates. The prompts are evaluated using BiomedCLIP to compute a fitness score ($s_j$). High-performing prompts are filtered to populate a buffer ($\mathcal{U}^{(t)}$) from which the next generation's candidates are selected. Finally, a crowding step is applied to the evolved population to yield a diverse, optimal set of prompts ($P^*$) for explainable diagnosis.
• Figure 2: Effect of prompt selection strategy on convergence. Our proposed Roulette Wheel selection achieves the best balance of exploration and exploitation, leading to more stable and consistent performance gains compared to Best-N and Random selection.
• Figure 3: Impact of generation size per iteration on performance. The 10-pair setting (our choice) provides the optimal balance, outperforming 5 pairs and 50 pairs.
• Figure 4: Effect of Chain-of-Thought (CoT) prompting, which consistently outperforms a direct-instruction baseline.
• Figure 5: Effect of initial population size on performance. A larger and more diverse initial population (50) consistently outperforms smaller sets