MedPI: Evaluating AI Systems in Medical Patient-facing Interactions

TL;DR

MedPI introduces a high-dimensional, end-to-end framework for evaluating AI systems in patient-facing conversations by integrating synthetic Patient Packets, memory-enabled AI Patients, a Task Matrix of encounter types, an Accreditation-aligned Evaluation Framework with $105$ dimensions, and AI Judges. Across $7{,}097$ conversations involving $9$ models and $366$ synthetic patients, the study reveals that even leading models struggle with core diagnostic reasoning and safety, despite strong reliability in other aspects. This platform enables scalable, granular stress-testing of clinical conversational capabilities and provides a reusable pipeline for continuous improvement, regulatory alignment, and education, with all artifacts openly released. The results highlight critical gaps in differential diagnosis and safety that must be addressed before real-world deployment, and they offer a concrete, multi-faceted roadmap for model development, evaluation, and governance in medical AI contexts.

Abstract

We present MedPI, a high-dimensional benchmark for evaluating large language models (LLMs) in patient-clinician conversations. Unlike single-turn question-answer (QA) benchmarks, MedPI evaluates the medical dialogue across 105 dimensions comprising the medical process, treatment safety, treatment outcomes and doctor-patient communication across a granular, accreditation-aligned rubric. MedPI comprises five layers: (1) Patient Packets (synthetic EHR-like ground truth); (2) an AI Patient instantiated through an LLM with memory and affect; (3) a Task Matrix spanning encounter reasons (e.g. anxiety, pregnancy, wellness checkup) x encounter objectives (e.g. diagnosis, lifestyle advice, medication advice); (4) an Evaluation Framework with 105 dimensions on a 1-4 scale mapped to the Accreditation Council for Graduate Medical Education (ACGME) competencies; and (5) AI Judges that are calibrated, committee-based LLMs providing scores, flags, and evidence-linked rationales. We evaluate 9 flagship models -- Claude Opus 4.1, Claude Sonnet 4, MedGemma, Gemini 2.5 Pro, Llama 3.3 70b Instruct, GPT-5, GPT OSS 120b, o3, Grok-4 -- across 366 AI Patients and 7,097 conversations using a standardized "vanilla clinician" prompt. For all LLMs, we observe low performance across a variety of dimensions, in particular on differential diagnosis. Our work can help guide future use of LLMs for diagnosis and treatment recommendations.

Figures (8)

• Figure 1: Overview of MedPI. Starting with a task matrix of encounter reason $\times$ objective, we generate synthetic patient records, which are used to generate a memory log that instantiate an LLM patient and an LLM doctor into a conversation aimed towards diagnosis or treatment. An evaluation framework based on a committee of LLM judges with tailored rubrics is used to evaluate the conversations across 105 dimensions.
• Figure 2: Mean normalized performance (0--100%) of each model across the seven MedPI competency meta-categories. Scores aggregate 105 rubric dimension and arre averaged over all relevant conversations per model.
• Figure 3: Model Performance Across Evaluation Dimension Groups Many models show low scores in differential diagnosis and high scores in model reliability.
• Figure 4: Percentage of dimensions with each score (1,2,3,4), by model. GPT-5 has the largest number of dimensions scoring either a 4 or 3.
• Figure 6: Normalized scores across all 105 metrics in MedPI. (continued on next page)