MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering

TL;DR

MedXplain-VQA advances explainable medical visual question answering by integrating domain-adapted vision-language modeling with medical query reformulation, enhanced visual grounding, precise region localization, and structured chain-of-thought reasoning via multi-modal LLMs. A novel medical-domain evaluation framework replaces traditional NLP metrics to assess terminology coverage, clinical structure, attention relevance, and reasoning confidence. On 500 PathVQA histopathology samples, the system achieves a composite score of $0.683$ and reasoning confidence of $0.890$, identifying 3–5 diagnostically relevant regions per sample and producing clinically coherent explanations averaging 57 words. The results underscore the value of systematic component integration—particularly medical query reformulation and chain-of-thought reasoning—for enhancing transparency and educational utility in clinical AI, while highlighting computational and validation challenges to be addressed for clinical deployment.

Abstract

Explainability is critical for the clinical adoption of medical visual question answering (VQA) systems, as physicians require transparent reasoning to trust AI-generated diagnoses. We present MedXplain-VQA, a comprehensive framework integrating five explainable AI components to deliver interpretable medical image analysis. The framework leverages a fine-tuned BLIP-2 backbone, medical query reformulation, enhanced Grad-CAM attention, precise region extraction, and structured chain-of-thought reasoning via multi-modal language models. To evaluate the system, we introduce a medical-domain-specific framework replacing traditional NLP metrics with clinically relevant assessments, including terminology coverage, clinical structure quality, and attention region relevance. Experiments on 500 PathVQA histopathology samples demonstrate substantial improvements, with the enhanced system achieving a composite score of 0.683 compared to 0.378 for baseline methods, while maintaining high reasoning confidence (0.890). Our system identifies 3-5 diagnostically relevant regions per sample and generates structured explanations averaging 57 words with appropriate clinical terminology. Ablation studies reveal that query reformulation provides the most significant initial improvement, while chain-of-thought reasoning enables systematic diagnostic processes. These findings underscore the potential of MedXplain-VQA as a robust, explainable medical VQA system. Future work will focus on validation with medical experts and large-scale clinical datasets to ensure clinical readiness.

Figures (4)

• Figure 1: MedXplain-VQA system architecture showing the five-stage progressive enhancement pipeline from input medical image and question to final explainable answer with visual evidence and reasoning chain.
• Figure 2: Multi-modal LLM integration process showing fusion of visual, spatial, textual, and reasoning modalities for unified explainable answer generation. The system employs Gemini 1.5-Pro for query reformulation and Gemini 1.5-Flash for multimodal integration, processing original images, attention heatmaps, bounding boxes, and chain-of-thought reasoning to produce comprehensive medical explanations.
• Figure 3: MedXplain-VQA explainability features comparison across system configurations. The radar chart demonstrates progressive enhancement in attention quality, reasoning confidence, medical terminology usage, clinical structure, and explanation coherence from basic to enhanced configurations.
• Figure 4: Enhanced MedXplain-VQA system demonstration on cardiovascular pathology. Shows: (a) Original histopathology image, (b) Bounding box detection identifying 5 regions with confidence scores 0.815-1.000, (c) Grad-CAM attention heatmap with color-coded intensity (red=highest attention, blue=lower relevance), and (d) Integrated visualization combining all explainability components.