From Conversation to Query Execution: Benchmarking User and Tool Interactions for EHR Database Agents

TL;DR

EHR-ChatQA, an interactive database question answering benchmark that evaluates the end-to-end workflow of database agents: clarifying user questions, using tools to resolve value mismatches, and generating correct SQL to deliver accurate answers is introduced.

Abstract

Despite the impressive performance of LLM-powered agents, their adoption for Electronic Health Record (EHR) data access remains limited by the absence of benchmarks that adequately capture real-world clinical data access flows. In practice, two core challenges hinder deployment: query ambiguity from vague user questions and value mismatch between user terminology and database entries. To address this, we introduce EHR-ChatQA, an interactive database question answering benchmark that evaluates the end-to-end workflow of database agents: clarifying user questions, using tools to resolve value mismatches, and generating correct SQL to deliver accurate answers. To cover diverse patterns of query ambiguity and value mismatch, EHR-ChatQA assesses agents in a simulated environment with an LLM-based user across two interaction flows: Incremental Query Refinement (IncreQA), where users add constraints to existing queries, and Adaptive Query Refinement (AdaptQA), where users adjust their search goals mid-conversation. Experiments with state-of-the-art LLMs (e.g., o4-mini and Gemini-2.5-Flash) over five i.i.d. trials show that while the best-performing agents achieve Pass@5 of over 90% (at least one of five trials) on IncreQA and 60-70% on AdaptQA, their Pass^5 (consistent success across all five trials) is substantially lower, with gaps of up to about 60%. These results underscore the need to build agents that are not only performant but also robust for the safety-critical EHR domain. Finally, we provide diagnostic insights into common failure modes to guide future agent development. Our code and data are publicly available at https://github.com/glee4810/EHR-ChatQA.

Figures (4)

• Figure 1: Overview of EHR-ChatQA. Our benchmark places an agent in a simulated environment with an LLM-powered user and tools. The agent must manage the entire workflow autonomously: clarifying vague user questions, resolving terminology mismatches, generating and executing accurate SQL, and returning the final answer. When the interaction is over, each trace is verified by a simulation validator, and only traces that pass proceed to rule-based evaluation.
• Figure 2: Example interaction trace in EHR-ChatQA (AdaptQA). Given a vague initial user question, the agent clarifies the request (1) by refining "some statistics" into the specific goal of "the percentage." When the initial query yields no matching records due to a value mismatch, the user guides the agent to search for its generic name instead (2). The agent then incorporates clinical knowledge retrieved via web_search() to reformulate the query (3), and finally executes the revised query to produce the correct answer (4).
• Figure 3: Detailed performance of the o4-mini-powered agent.
• Figure 4: Distribution of the difference between the maximum and minimum number of user turns for each unique task in IncreQA and AdaptQA. A unique task is defined by db_id, task_type, and task_id. Most tasks have a difference of 1 to 3 turns, which indicates that the user simulator generates stochastic conversations rather than fixed scripts.