MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents

TL;DR

MedAgentBench tackles the paucity of benchmarks for evaluating LLM agents in medical contexts by delivering a realistic, interactive EMR environment and a $300$-task, $100$-patient dataset with over $700{,}000$ data elements. The framework combines a FHIR-based interface, a simple agent orchestrator, and an evaluation protocol focusing on task success under a strict $1$-shot (pass@1) regime across $11$ models, with Claude 3.5 Sonnet v2 achieving the top overall score of $69.67$ in a challenging, safety-conscious setting. The study reveals that information-retrieval tasks are generally easier for current agents than action-based EMR modifications, underscoring gaps in planning and tool-use capabilities needed for reliable clinical automation. Overall, MedAgentBench offers a scalable, publicly available platform to drive progress in producing safe, effective agentic systems for real-world clinical workflows.

Abstract

Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents thereby surpassing their traditional role as chatbots. These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level. However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making the evaluation of LLMs on complex tasks in interactive healthcare environments challenging. To address this gap, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of large language models within medical records contexts. MedAgentBench encompasses 300 patient-specific clinically-derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a FHIR-compliant interactive environment, and an accompanying codebase. The environment uses the standard APIs and communication infrastructure used in modern EMR systems, so it can be easily migrated into live EMR systems. MedAgentBench presents an unsaturated agent-oriented benchmark that current state-of-the-art LLMs exhibit some ability to succeed at. The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%. However, there is still substantial space for improvement which gives the community a next direction to optimize. Furthermore, there is significant variation in performance across task categories. MedAgentBench establishes this and is publicly available at https://github.com/stanfordmlgroup/MedAgentBench , offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of large language models within the medical domain.

Figures (3)

• Figure 1: Schematic diagram of MedAgentBench framework. The MedAgentBench workflow begins with a clinician specifying a high-level task, after which the agent orchestrator interacts with both the LLM provider and the electronic medical record environment to finish the task and finally provide feedback to the clinician.
• Figure 2: Example successful trajectory and common error patterns in MedAgentBench. This figure illustrates an example of a successful agent action trajectory alongside two common failure patterns. (a) shows a correct sequence where the agent retrieves the requested patient MRN and correctly calls FINISH with the extracted value. (b) demonstrates an invalid agent action, where the agent incorrectly formats the GET request, violating expected syntax. (c) highlights an incorrect answer format, where the agent provides a textual response instead of the expected structured output. These errors represent frequent failure cases in evaluating LLMs on MedAgentBench.
• Figure 3: Screenshot of frontend of the FHIR-compliant interactive environment.