SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering

TL;DR

SCARE introduces a unified post-hoc verification benchmark for EHR QA that jointly evaluates question answerability classification and SQL correction/correction validation. By grounding 4,200 question–SQL–answer triples in MIMIC-III, MIMIC-IV, and eICU and leveraging seven diverse text-to-SQL models, the paper reveals a persistent trade-off between correctly classifying problematic questions and preserving or correcting valid queries. The results show iterative refinement and hybrid methods offer the strongest balance, yet nuanced ambiguities remain difficult to detect, underscoring the need for robust, auditable verification in clinical deployments. Overall, SCARE provides a diagnostic framework to drive safe integration of LLMs in live EHR QA systems and guide future research on reliable post-hoc safety mechanisms.

Abstract

Recent advances in Large Language Models (LLMs) have enabled the development of text-to-SQL models that allow clinicians to query structured data stored in Electronic Health Records (EHRs) using natural language. However, deploying these models for EHR question answering (QA) systems in safety-critical clinical environments remains challenging: incorrect SQL queries-whether caused by model errors or problematic user inputs-can undermine clinical decision-making and jeopardize patient care. While prior work has mainly focused on improving SQL generation accuracy or filtering questions before execution, there is a lack of a unified benchmark for evaluating independent post-hoc verification mechanisms (i.e., a component that inspects and validates the generated SQL before execution), which is crucial for safe deployment. To fill this gap, we introduce SCARE, a benchmark for evaluating methods that function as a post-hoc safety layer in EHR QA systems. SCARE evaluates the joint task of (1) classifying question answerability (i.e., determining whether a question is answerable, ambiguous, or unanswerable) and (2) verifying or correcting candidate SQL queries. The benchmark comprises 4,200 triples of questions, candidate SQL queries, and expected model outputs, grounded in the MIMIC-III, MIMIC-IV, and eICU databases. It covers a diverse set of questions and corresponding candidate SQL queries generated by seven different text-to-SQL models, ensuring a realistic and challenging evaluation. Using SCARE, we benchmark a range of approaches-from two-stage methods to agentic frameworks. Our experiments reveal a critical trade-off between question classification and SQL error correction, highlighting key challenges and outlining directions for future research.

Figures (2)

• Figure 1: Overview of the SCARE benchmark for evaluating a post-hoc verification layer. The task assumes a candidate SQL query has already been generated by an upstream (and potentially black-box) text-to-SQL model. The layer then takes both the user's question and the candidate SQL as input to decide one of four actions: (1) for an answerable question with correct SQL, preserve the query; (2) for an answerable question with incorrect SQL, correct the query; (3) for an ambiguous question, identify the ambiguity for user clarification; and (4) for an unanswerable question, reject the query and inform the user, thus ensuring the overall reliability and safety of the EHR QA system.
• Figure 2: Overview of the SCARE benchmark construction pipeline.