Advancing Conversational Diagnostic AI with Multimodal Reasoning

TL;DR

The paper addresses the gap in diagnostic AI by enabling multimodal reasoning within conversational clinicians, implementing AMIE with a state-aware mechanism that dynamically gathers and interprets artifacts such as skin images, ECGs, and clinical documents. It combines a three-phase dialogue framework (History Taking, Diagnosis & Management, Follow-up) with a post-dialogue structured summary and an automated evaluation pipeline, validated through a blinded, OSCE-style study against PCPs across 105 scenarios. Automated perception tests and ablation analyses demonstrate the value of state-aware reasoning and history-taking in improving diagnostic accuracy and robustness, while expert OSCE results show AMIE often outperforming PCPs on multimodal understanding, artifact reasoning, and patient-centered communication. The work highlights significant progress for telemedicine AI while acknowledging the need for real-world clinical validation, safety considerations, and equity across diverse patient populations. Overall, the approach demonstrates a practical pathway to integrate multimodal data into AI-guided telehealth while maintaining clinical standards and interpretability.

Abstract

Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in medical consultation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process that emulates experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors. We constructed 105 evaluation scenarios using artifacts like smartphone skin photos, ECGs, and PDFs of clinical documents across diverse conditions and demographics. Our rubric assessed multimodal capabilities and other clinically meaningful axes like history-taking, diagnostic accuracy, management reasoning, communication, and empathy. Specialist evaluation showed AMIE to be superior to PCPs on 7/9 multimodal and 29/32 non-multimodal axes (including diagnostic accuracy). The results show clear progress in multimodal conversational diagnostic AI, but real-world translation needs further research.

Figures (29)

• Figure 1: Overview of key contributions. This figure provides a schematic overview of the key components enabling and evaluating multimodal diagnostic conversations within the Articulate Medical Intelligence Explorer (AMIE) system, facilitated through a multimodal chat interface. The main aspects include A. Multimodal state-aware reasoning: AMIE employs a novel state-aware dialogue phase transition framework built on the publicly available Gemini 2.0 Flash model. This dynamically controls the conversation flow through history taking, diagnosis, and management phases, guided by intermediate outputs reflecting patient state and diagnostic uncertainty, allowing strategic requests and interpretation of multimodal artifacts. B. Simulation environment: A comprehensive simulation framework enables rapid development and automated evaluation. It involves generating realistic patient scenarios grounded in real images and metadata, using Gemini 2.0 Flash, simulating turn-by-turn multimodal dialogues between AMIE and patient agents, and utilizing an auto-rater agent for assessment against clinical criteria. C. Randomized comparative study: AMIE's performance was rigorously evaluated against primary care physicians (PCPs) in a randomized, double-blind, OSCE-style study. Trained patient actors conducted synchronous chat consultations based on 105 diverse multimodal scenarios involving artifacts such as skin photos, ECGs, and clinical documents. D. Evaluation results: Specialist evaluations demonstrated that AMIE attains comparable or superior performance to PCPs in handling and reasoning about multimodal data over multiple evaluation axes alongside strong performance in diagnostic accuracy and overall consultation quality.
• Figure 2: Multimodal state-aware reasoning at inference. AMIE's state-aware dialogue phase transition framework, which structures the diagnostic conversation is illustrated here. The system progresses through three distinct phases, each with a specific goal: (1) History Taking (gathering comprehensive patient information), (2) (Differential) Diagnosis & Management (formulating and presenting DDx and Mx Plan), and (3) Answer Follow-up Questions (addressing remaining concerns). Within each phase, AMIE maintains an internal state – its dynamic understanding of the patient's situation, evolving diagnoses (DDx), and knowledge gaps – derived from the dialogue history and any multimodal inputs. This state guides specific actions, such as asking targeted questions, requesting multimodal data (if needed), generating internal summaries/DDx, or providing explanations. Transitions between phases are triggered automatically when the system assesses that the objectives of the current phase (e.g., sufficient information gathered, DDx presented) have been met, based on its internal state evaluation. This mechanism enforces a structured yet flexible dialogue flow, inspired by the methodical approach of experienced clinicians.
• Figure 3: Overview of multimodal dialogue simulation and evaluation framework. This figure illustrates the three key components of the system. Step 1 (Patient Scenario Generation): Comprehensive patient profiles are created, including condition, demographics, symptoms, and medical history, using information derived from web searches and real-world datasets e.g., PTB-XL (for cardiology) and SCIN (for dermatology). These profiles are then used to generate detailed patient scenarios, outlining the patient's presentation and expectations. Step 2 (Dialogue Simulation): A doctor agent and a patient agent engage in a text-based consultation with multimodal artifact upload. The doctor agent is instructed to provide empathetic and clinically accurate responses, while the patient agent responds truthfully based on the generated scenario. Step 3 (Auto-rating): An auto-rater agent evaluates the simulated dialogue based on pre-defined criteria, including management appropriateness, information gathering effectiveness, and the presence of hallucinations. Qualitative feedback is also provided by the auto-rater to explain the scores.
• Figure 4: Overview of expert evaluation (OSCE). Step 1: Patient actors simulate cases based on multimodal scenarios containing history, symptoms, and visual artefacts (skin photos, ECGs, or clinical documents) and conduct synchronous chat consultations with both a PCP and AMIE in a randomized, blinded order. Step 2 (Evaluation): After consultations, patient actors assess interaction quality by filling out a questionnaire, providing patient-centric quality measures. In parallel, both PCPs and AMIE document their findings by generating answers to a separate post-questionnaire detailing differential diagnosis, management plans (investigation, treatment and escalation needs) and also salient image findings. Subsequently, specialist physicians evaluate the performance of PCPs and AMIE based on the dialogue transcript, post-questionnaire answers, and scenario ground truth across multiple criteria.
• Figure 5: Comparison of OSCE metrics between PCPs and AMIE. Across all three panels, the error bars represent $95\%$ confidence intervals estimated based on $10^4$ bootstrap samples. A. Top-k differential diagnosis (DDx) accuracy. Both AMIE and PCPs have the opportunity to submit a differential diagnosis list (at least 3, up to 10 plausible items, ordered by likelihood) for each dialogue. We compute the top-k accuracy using an auto-rater that compares each of the 10 diagnoses with the ground truth. AMIE outperforms PCPs in diagnostic accuracy on average ($p < 0.001$). B. Subgroup analysis of DDx accuracy. We compare the top-3 accuracy across subgroups defined by three axes defined in the MUH rubric: (1) image quality, (2) image grounded reasoning and (3) hallucination of image findings in consultation (see Table. \ref{['tab:mm_osce_rubric']} for details). These findings provide some insight into why AMIE may be more accurate. C. Distributions of relative performance across patient scenarios. In particular, we present the proportion of scenarios where PCPs are rated more favourably than AMIE and vice versa for three categories of OSCE rubric, namely multimodal understanding and handling rubric (right), non-multimodal specialist metrics (middle) and non-multimodal patient-centric metrics (right). Each specialist and patient actor separately rated two dialogues (one with PCP and the other with AMIE) for the same scenario, and their relative ranking is derived from their scores. For the specialist metrics, each scenario was evaluated by three different expert physicians, yielding three pairwise rankings which we aggregate by taking the majority vote. We assigned the 'Tie' label to cases where two or more specialists assigned the same ratings to both PCP and AMIE consultations or the cases where the opinions on the relative performance were equally divided. Significant differences between AMIE and PCP segments calculated using a one-sided chi-squared test are indicated by asterisks ($*:p<0.05$, $**:p<0.01$, $***:p<0.001$, $n.s.:$ not significant). Figure \ref{['fig:osce_summary_results_distribution']} in the Appendix shows the underlying distributions of ratings for the same set of metrics.