Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering

Gyubok Lee, Elea Bach, Eric Yang, Tom Pollard, Alistair Johnson, Edward Choi, Yugang jia, Jong Ha Lee

TL;DR

The paper introduces FHIR-AgentBench, a realism-grounded benchmark for evaluating LLM agents on interoperable EHR QA using HL7 FHIR, grounded in 2,931 clinician-sourced questions over de-identified MIMIC-IV-FHIR data. It defines a two-stage retrieval-and-answer framework and systematic ablations across data retrieval strategies, interaction patterns, and reasoning modes (natural language vs code). The results show that current state-of-the-art agents struggle with FHIR's graph-like structure, with best-performing multi-turn agents achieving only about 50% answer correctness, highlighting substantial bottlenecks in retrieval precision and interpretation of FHIR resources. The authors also release the dataset, conversion tooling, and evaluation suite to foster reproducibility and tool development for robust, interoperable clinical AI.

Abstract

The recent shift toward the Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) standard opens a new frontier for clinical AI, demanding LLM agents to navigate complex, resource-based data models instead of conventional structured health data. However, existing benchmarks have lagged behind this transition, lacking the realism needed to evaluate recent LLMs on interoperable clinical data. To bridge this gap, we introduce FHIR-AgentBench, a benchmark that grounds 2,931 real-world clinical questions in the HL7 FHIR standard. Using this benchmark, we systematically evaluate agentic frameworks, comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation). Our experiments highlight the practical challenges of retrieving data from intricate FHIR resources and the difficulty of reasoning over them, both of which critically affect question answering performance. We publicly release the FHIR-AgentBench dataset and evaluation suite (https://github.com/glee4810/FHIR-AgentBench) to promote reproducible research and the development of robust, reliable LLM agents for clinical applications.

FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering

TL;DR

The paper introduces FHIR-AgentBench, a realism-grounded benchmark for evaluating LLM agents on interoperable EHR QA using HL7 FHIR, grounded in 2,931 clinician-sourced questions over de-identified MIMIC-IV-FHIR data. It defines a two-stage retrieval-and-answer framework and systematic ablations across data retrieval strategies, interaction patterns, and reasoning modes (natural language vs code). The results show that current state-of-the-art agents struggle with FHIR's graph-like structure, with best-performing multi-turn agents achieving only about 50% answer correctness, highlighting substantial bottlenecks in retrieval precision and interpretation of FHIR resources. The authors also release the dataset, conversion tooling, and evaluation suite to foster reproducibility and tool development for robust, interoperable clinical AI.

Abstract

The recent shift toward the Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) standard opens a new frontier for clinical AI, demanding LLM agents to navigate complex, resource-based data models instead of conventional structured health data. However, existing benchmarks have lagged behind this transition, lacking the realism needed to evaluate recent LLMs on interoperable clinical data. To bridge this gap, we introduce FHIR-AgentBench, a benchmark that grounds 2,931 real-world clinical questions in the HL7 FHIR standard. Using this benchmark, we systematically evaluate agentic frameworks, comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation). Our experiments highlight the practical challenges of retrieving data from intricate FHIR resources and the difficulty of reasoning over them, both of which critically affect question answering performance. We publicly release the FHIR-AgentBench dataset and evaluation suite (https://github.com/glee4810/FHIR-AgentBench) to promote reproducible research and the development of robust, reliable LLM agents for clinical applications.

Paper Structure

This paper contains 40 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. SQL-based EHR QA benchmarks
  4. FHIR-integrated QA benchmarks
  5. Agent-based clinical benchmarks and evaluation
  6. Benchmark Construction
  7. FHIR Compatible Question Selection
  8. Natural Language Question and Answer Generation
  9. Natural Language Question-Answer Validation
  10. FHIR Resource ID Generation
  11. Dataset Composition and Statistics
  12. Experiments
  13. Evaluation Framework
  14. Agent Architectures
  15. Implementation Details
  16. ...and 25 more sections

Figures (5)

  • Figure 1: Overview of FHIR-AgentBench.
  • Figure 2: Overview of the construction process for FHIR-AgentBench.
  • Figure 3: Distribution of questions by the FHIR resource type required to answer them. Observation includes input and output events, lab events, chart events, and microbiology, which constitutes the largest portion.
  • Figure 4: Relative gain in answer correctness across precision buckets for the o4-mini model under different agent frameworks. Gains are calculated after filtering to questions with perfect recall, isolating the effect of unnecessary resources on answer correctness.
  • Figure B1: Illustration of FHIR resource ID translation process from SQL query.