Biomedical Nested NER with Large Language Model and UMLS Heuristics

TL;DR

This work tackles nested biomedical named entity recognition within the BioNNE English track, aiming to identify eight entity types in PubMed abstracts. It proposes a hybrid pipeline that combines Mixtral 8x7B instruct as a general-purpose LLM, ScispaCy NER for DISEASE and CHEM detection, and UMLS-based semantic-type heuristics to finalize entity categorization, augmented by acronym detection. The approach achieves an internal validation F1 of $0.39$ and a test F1 of $0.348$, with ablation showing UMLS heuristics substantially boosting precision by reducing false positives; without heuristics, the macro-F1 drops to $0.2151$. The study demonstrates the viability of leveraging LLMs with domain-specific rules for biomedical NER, while highlighting limitations in handling outer nested entities and context-sensitive category distinctions, and outlines concrete paths for fine-tuning and bilingual extension to improve performance.

Abstract

In this paper, we present our system for the BioNNE English track, which aims to extract 8 types of biomedical nested named entities from biomedical text. We use a large language model (Mixtral 8x7B instruct) and ScispaCy NER model to identify entities in an article and build custom heuristics based on unified medical language system (UMLS) semantic types to categorize the entities. We discuss the results and limitations of our system and propose future improvements. Our system achieved an F1 score of 0.39 on the BioNNE validation set and 0.348 on the test set.

Figures (3)

• Figure 1: BioNNE System Design
• Figure 2: BioNNE Data Flow
• Figure 3: Example Prompt and Response for ANATOMY entity