PatientVLM Meets DocVLM: Pre-Consultation Dialogue Between Vision-Language Models for Efficient Diagnosis

TL;DR

This work introduces the Pre-Consultation Dialogue Framework (PCDF) to address the limitations of image-only diagnosis by enabling dialogue-driven reasoning in vision–language models. PCDF pairs a DocVLM (doctor) and a PatientVLM (patient) to simulate realistic, visually grounded doctor–patient conversations, generating image–dialogue–diagnosis triplets that are then used to finetune the DocVLM. Across four medical imaging benchmarks, PCDF yields consistent diagnostic gains over image-only baselines and other VLMs, with an average F1 improvement of $11.48$ and notable gains when dialogue length increases. Preliminary clinical validation further suggests the synthetic dialogues are clinically relevant, comprehensive, and reasonably realistic, supporting the potential of dialogue-conditioned supervision to improve interpretability and deployment readiness in medical AI.

Abstract

Traditionally, AI research in medical diagnosis has largely centered on image analysis. While this has led to notable advancements, the absence of patient-reported symptoms continues to hinder diagnostic accuracy. To address this, we propose a Pre-Consultation Dialogue Framework (PCDF) that mimics real-world diagnostic procedures, where doctors iteratively query patients before reaching a conclusion. Specifically, we simulate diagnostic dialogues between two vision-language models (VLMs): a DocVLM, which generates follow-up questions based on the image and dialogue history, and a PatientVLM, which responds using a symptom profile derived from the ground-truth diagnosis. We additionally conducted a small-scale clinical validation of the synthetic symptoms generated by our framework, with licensed clinicians confirming their clinical relevance, symptom coverage, and overall realism. These findings indicate that the resulting DocVLM-PatientVLM interactions form coherent, multi-turn consultations paired with images and diagnoses, which we then use to fine-tune the DocVLM. This dialogue-based supervision leads to substantial gains over image-only training, highlighting the value of realistic symptom elicitation for diagnosis.

Figures (5)

• Figure 1: Overview of the Pre-Consultation Dialogue Framework (PCDF). (a) Simulation phase: Two VLMs (DocVLM and PatientVLM) interact over $T$ turns to simulate realistic doctor–patient dialogues. (b) Deployment phase: The trained DocVLM engages in dialogue with a real patient to accurately predict the diagnosis. (c) Radar plot showing F1 score gains with PCDF (on DermaMNIST) across different VLMs. (Best viewed in color).
• Figure 2: The Pre-Consultation Dialogue Framework (PCDF). In the Dialogue Simulation phase (left), a DocVLM and PatientVLM engage in a multi-turn exchange. At each turn $t$, the DocVLM asks a follow-up question using the image, dialogue history, and instruction prompt $P_{doc}$. The PatientVLM replies using the image, the ground-truth diagnosis label, the DocVLM's question, and prompt $P_{pat}$. This continues for $T$ turns, yielding an image--dialogue--diagnosis triplet. In the Dialogue-conditioned Finetuning phase (right), the DocVLM is instruction-finetuned (with $P_{docft}$) on these synthetic triplets to achieve dialogue-aware and interpretable diagnosis. (Best viewed in color.)
• Figure 3: A selection of dialogues generated between DocVLM and PatientVLM.
• Figure 5: Additional samples of dialogues generated between DocVLM and PatientVLM.
• Figure 6: A selection of diagnostic predictions from MedGemma3-4B across three settings: zero-shot, image-only fine-tuned, and PCDF-enabled. PCDF consistently achieves accurate diagnoses (shown in green) while the same model under zero-shot and image-only fine-tuned settings frequently misclassify the diagnosis (shown in red), demonstrating the effectiveness of PCDF-enabled dialogue-driven diagnostic reasoning.