Named Clinical Entity Recognition Benchmark

Abstract

This technical report introduces a Named Clinical Entity Recognition Benchmark for evaluating language models in healthcare, addressing the crucial natural language processing (NLP) task of extracting structured information from clinical narratives to support applications like automated coding, clinical trial cohort identification, and clinical decision support. The leaderboard provides a standardized platform for assessing diverse language models, including encoder and decoder architectures, on their ability to identify and classify clinical entities across multiple medical domains. A curated collection of openly available clinical datasets is utilized, encompassing entities such as diseases, symptoms, medications, procedures, and laboratory measurements. Importantly, these entities are standardized according to the Observational Medical Outcomes Partnership (OMOP) Common Data Model, ensuring consistency and interoperability across different healthcare systems and datasets, and a comprehensive evaluation of model performance. Performance of models is primarily assessed using the F1-score, and it is complemented by various assessment modes to provide comprehensive insights into model performance. The report also includes a brief analysis of models evaluated to date, highlighting observed trends and limitations. By establishing this benchmarking framework, the leaderboard aims to promote transparency, facilitate comparative analyses, and drive innovation in clinical entity recognition tasks, addressing the need for robust evaluation methods in healthcare NLP.

Figures (8)

• Figure 1: Overall performance of models across six clinical entities. Box plots represent F1-scores of various models across the clinical entity types for each metric approach: token-based (left), and span-based. Each dots represent the performance of a model.
• Figure 2: Performance across model sizes and architectures. Both the plots represent the span based F1 scores. For size comparision (left) the average F1 score across clinical entities is used. For architecture comparision (right) only decoder and GLiNER encoder models are used. Additionally, closed source models are filtered out.
• Figure 3: Effect of Training. Span based metrics of the open-source decoder models from the leaderboard are used here.
• Figure 4: Model rankings according to token-based and span-based F1-scores. The overall (average) token (left) and span-based metrics for each model are shown. The top-4 performing models, according to the span-based F1-score, are highlighted in orange, and the performance of GPT-4o is shown in teal. Models with overall performances below 20% are not shown.
• Figure 5: Data Distribution of Clinical Entities