LLM-based Prompt Ensemble for Reliable Medical Entity Recognition from EHRs

TL;DR

This work tackles the challenge of extracting structured medical entities from unstructured EHR text using prompt-based learning with LLMs. It compares GPT-4o and DeepSeek-R1 under zero-shot, few-shot, and a prompt ensemble that aggregates multiple prompts through embedding-based reconciliation and majority voting. The GPT-4o ensemble achieves a $F1$-score of 0.95 with recall 0.98 in classification, outperforming DeepSeek-R1 and demonstrating improved reliability over single-prompt approaches. The results indicate that careful prompt design combined with cross-prompt ensembling can enable robust clinical NER without fine-tuning, offering scalable, safer extraction of key clinical entities for downstream applications.

Abstract

Electronic Health Records (EHRs) are digital records of patient information, often containing unstructured clinical text. Named Entity Recognition (NER) is essential in EHRs for extracting key medical entities like problems, tests, and treatments to support downstream clinical applications. This paper explores prompt-based medical entity recognition using large language models (LLMs), specifically GPT-4o and DeepSeek-R1, guided by various prompt engineering techniques, including zero-shot, few-shot, and an ensemble approach. Among all strategies, GPT-4o with prompt ensemble achieved the highest classification performance with an F1-score of 0.95 and recall of 0.98, outperforming DeepSeek-R1 on the task. The ensemble method improved reliability by aggregating outputs through embedding-based similarity and majority voting.

Figures (4)

• Figure 1: Example of entities in EHR
• Figure 2: Components of Prompting Template
• Figure 3: Workflow of the Prompt Ensemble
• Figure 4: Prompt Execution Time