Human or LLM as Standardized Patients? A Comparative Study for Medical Education

TL;DR

This paper proposes EasyMED, a multi-agent framework for virtual Standardized Patient (SP) training that separates patient simulation, intent recognition, and evaluation into coordinated components. It also introduces SPBench, a benchmark built from authentic SP–student dialogues across 14 specialties and eight evaluation criteria, enabling reproducible, turn- and session-level assessment. In a four-week controlled crossover study with undergraduate medical students, EasyMED delivered learning outcomes comparable to traditional human SP training while offering greater flexibility, psychological safety, and substantial cost reductions, with pronounced benefits for lower-baseline learners. The work demonstrates the feasibility and pedagogical value of multi-agent, LLM-based SP systems for scalable, high-quality clinical education and provides a path toward widespread, cost-effective skills training in medical education.

Abstract

Standardized Patients (SP) are indispensable for clinical skills training but remain expensive, inflexible, and difficult to scale. Existing large-language-model (LLM)-based SP simulators promise lower cost yet show inconsistent behavior and lack rigorous comparison with human SP. We present EasyMED, a multi-agent framework combining a Patient Agent for realistic dialogue, an Auxiliary Agent for factual consistency, and an Evaluation Agent that delivers actionable feedback. To support systematic assessment, we introduce SPBench, a benchmark of real SP-doctor interactions spanning 14 specialties and eight expert-defined evaluation criteria. Experiments demonstrate that EasyMED matches human SP learning outcomes while producing greater skill gains for lower-baseline students and offering improved flexibility, psychological safety, and cost efficiency.

Figures (14)

• Figure 1: The multi-agent architecture of our virtual medical training system. This "division of labor" model includes: (1) a Patient Agent to simulate patient dialogue, (2) an Intention Identification Agent to interpret the user's clinical queries, and (3) an Evaluation Agent to assess performance against ground truth.
• Figure 2: Distribution of clinical cases in the SPBench dataset by medical department.
• Figure 3: Demographic distribution of cases in the SPBench dataset. The left panel shows the age distribution, and the right panel shows the gender distribution.
• Figure 4: Boxplots comparing the baseline scores of Group A and Group B. Each circle represents the score of an individual participant. The similarity in the distributions demonstrates that the two groups were comparable at the start of the experiment.
• Figure 5: Learning trajectories of individual students and group averages for Group A (left) and Group B (right) across three assessment points. Each colored line tracks an individual student's performance, while the bold dashed line represents the group's average score.