ProMRVL-CAD: Proactive Dialogue System with Multi-Round Vision-Language Interactions for Computer-Aided Diagnosis

TL;DR

ProMRVL-CAD tackles the gap in medical AI dialogue by enabling proactive, multi-round interactions that jointly leverage medical images and patient history. It introduces two synergistic components: Pro-Q Gen for proactive questioning guided by a clinical knowledge graph, and MVP-DR Gen for multi-view report generation using a frozen LLM enhanced by a vision-language alignment layer and a joint loss $L = L_{classification} + \alpha L_{report}$. A synthetic ProDial dialogue dataset complements real clinical data to train the system and emulate doctor-patient consultations. On MIMIC-CXR and IU-Xray, ProMRVL-CAD achieves state-of-the-art report quality and robust diagnostic performance, while demonstrating resilience to low image quality and scalability via LoRA-based parameter reduction. The work also provides a first synthetic proactive dialogue resource blending visuals and textual health information, with practical implications for improving diagnostic communication and clinical workflow.

Abstract

Recent advancements in large language models (LLMs) have demonstrated extraordinary comprehension capabilities with remarkable breakthroughs on various vision-language tasks. However, the application of LLMs in generating reliable medical diagnostic reports remains in the early stages. Currently, medical LLMs typically feature a passive interaction model where doctors respond to patient queries with little or no involvement in analyzing medical images. In contrast, some ChatBots simply respond to predefined queries based on visual inputs, lacking interactive dialogue or consideration of medical history. As such, there is a gap between LLM-generated patient-ChatBot interactions and those occurring in actual patient-doctor consultations. To bridge this gap, we develop an LLM-based dialogue system, namely proactive multi-round vision-language interactions for computer-aided diagnosis (ProMRVL-CAD), to generate patient-friendly disease diagnostic reports. The proposed ProMRVL-CAD system allows proactive dialogue to provide patients with constant and reliable medical access via an integration of knowledge graph into a recommendation system. Specifically, we devise two generators: a Proactive Question Generator (Pro-Q Gen) to generate proactive questions that guide the diagnostic procedure and a Multi-Vision Patient-Text Diagnostic Report Generator (MVP-DR Gen) to produce high-quality diagnostic reports. Evaluating two real-world publicly available datasets, MIMIC-CXR and IU-Xray, our model has better quality in generating medical reports. We further demonstrate the performance of ProMRVL achieves robust under the scenarios with low image quality. Moreover, we have created a synthetic medical dialogue dataset that simulates proactive diagnostic interactions between patients and doctors, serving as a valuable resource for training LLM.

Figures (6)

• Figure 1: High-level comparisons between existing works and ProMRVL-CAD when medical images are necessary for the diagnosis/treatment. a): Question-Answering (QA) task; b): Visual QA task; c): ProMRVL-CAD (Ours). By supporting multi-round text-visual inputs, ProMRVL-CAD mimics the consultation process between doctors and patients to collect diagnosis-orientated information. In contrast, existing frameworks simply use the first visual for report generation and then passively answer questions from the patients (QA) or only propose questions related to the input visuals (VQA).
• Figure 2: The architecture of ProMRVL-CAD. It consists of two modules: Pro-Q Gen to prompt patients to provide more informative inputs and MVP-DR Gen to generate diagnostic reports from multi-modality inputs.
• Figure 3: An illustration of the knowledge graph with edge widths indicating the correlation between a disease and a symptom.
• Figure 4: Examples of synthetic proactive medical dialogue (training data) generated using medical history and medical images. The clinical concepts (highlighted by blue, orange, and green colors) are consistent among the medical history, the report, and the synthetic dialogue. Our synthetic dialogue is in line with the medical visuals and contains complementary information on health conditions to assist in disease diagnosis.
• Figure 5: Representative samples of the proactive dialogue produced by our proposed Pro-Q Gen. (a). A sample dialogue from a patient with weight loss and pneumonia. (b). A sample dialogue from a patient with several symptoms and various medical histories. Our proposed Pro-Q Gen could proactively pose queries to efficiently collect disease symptoms and medical history from the patients.