EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records

TL;DR

EHRCon provides a large-scale, clinically grounded dataset to detect inconsistencies between unstructured notes and structured EHR tables, using two schema variants (MIMIC-III and OMOP-CDM). It introduces CheckEHR, an eight-stage, LLM-based reasoning framework that decomposes the task into segmentation, entity recognition, time filtering, and SQL query generation to verify note-table alignment. Empirical results show meaningful gains in recall with few-shot prompting, but highlight persistent challenges in NER and complex schema interpretation. The work offers a path toward automated, scalable data quality assurance in healthcare documentation and supports cross-institution generalization through dual-schema evaluation.

Abstract

Electronic Health Records (EHRs) are integral for storing comprehensive patient medical records, combining structured data (e.g., medications) with detailed clinical notes (e.g., physician notes). These elements are essential for straightforward data retrieval and provide deep, contextual insights into patient care. However, they often suffer from discrepancies due to unintuitive EHR system designs and human errors, posing serious risks to patient safety. To address this, we developed EHRCon, a new dataset and task specifically designed to ensure data consistency between structured tables and unstructured notes in EHRs. EHRCon was crafted in collaboration with healthcare professionals using the MIMIC-III EHR dataset, and includes manual annotations of 4,101 entities across 105 clinical notes checked against database entries for consistency. EHRCon has two versions, one using the original MIMIC-III schema, and another using the OMOP CDM schema, in order to increase its applicability and generalizability. Furthermore, leveraging the capabilities of large language models, we introduce CheckEHR, a novel framework for verifying the consistency between clinical notes and database tables. CheckEHR utilizes an eight-stage process and shows promising results in both few-shot and zero-shot settings. The code is available at https://github.com/dustn1259/EHRCon.

Figures (24)

• Figure 1: Examples of consistent and inconsistent data between clinical notes and EHR tables: An inconsistent example (datetime) is when a clinical note records an HR (abbreviation for heart rate) of 167 on "2022-02-02" but the EHR table shows the same HR on "2022-02-06". A consistent example is when both the clinical note and the EHR table document the administration of Atorvastatin with matching drug name, dosage, and unit. Another example of inconsistency occurs when a clinical note mentions a hypertension diagnosis, but the EHR table lacks this information.
• Figure 2: Annotation process of EHRCon: The annotation process involves annotators reviewing clinical notes, identifying and classifying entities into Type 1 and Type 2, and extracting relevant information to generate and execute SQL queries. If the SQL queries yield no results, conditions (e.g., value or time) are masked to pinpoint where the inconsistency occurred. When annotators encounter corner cases, they update the labeling instructions through discussion. After all labeling is complete, a post-processing phase is conducted to ensure high-quality data.
• Figure 3: Overview of CheckEHR. The framework consists of eight distinct stages: Note Segmentation, Named Entity Recognition, Time Filtering, Table Identification, Pseudo Table Creation, Self-Correction, Value Reformatting, and Query Generation.
• Figure 4: Distribution of different note categories in MIMIC-III.
• Figure 5: Overall process of table preparation.