LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty

TL;DR

LinkNER addresses the instability of NER in open environments by coupling a fine-tuned local SpanNER with a black-box LLM via a Recognition-Detection-Classification (RDC) mechanism driven by uncertainty estimates. Uncertain spans from the local model are sent to the LLM for reclassification, and results are fused to yield robust entity recognition across standard, OOV, and OOD datasets. The study analyzes multiple uncertainty methods, LLM choices, and in-context learning strategies, offering practical guidance for web-related NER applications. Public code is provided, highlighting the method's feasibility for real-world deployment and further research.

Abstract

Named Entity Recognition (NER) serves as a fundamental task in natural language understanding, bearing direct implications for web content analysis, search engines, and information retrieval systems. Fine-tuned NER models exhibit satisfactory performance on standard NER benchmarks. However, due to limited fine-tuning data and lack of knowledge, it performs poorly on unseen entity recognition. As a result, the usability and reliability of NER models in web-related applications are compromised. Instead, Large Language Models (LLMs) like GPT-4 possess extensive external knowledge, but research indicates that they lack specialty for NER tasks. Furthermore, non-public and large-scale weights make tuning LLMs difficult. To address these challenges, we propose a framework that combines small fine-tuned models with LLMs (LinkNER) and an uncertainty-based linking strategy called RDC that enables fine-tuned models to complement black-box LLMs, achieving better performance. We experiment with both standard NER test sets and noisy social media datasets. LinkNER enhances NER task performance, notably surpassing SOTA models in robustness tests. We also quantitatively analyze the influence of key components like uncertainty estimation methods, LLMs, and in-context learning on diverse NER tasks, offering specific web-related recommendations. Code is available at https://github.com/zhzhengit/LinkNER.

Figures (7)

• Figure 1: Illustration of LinkNER processing the NER task. If the local NER model is uncertain about its output (see the second case), the linked LLM undergoes further classification to determine the final result.
• Figure 2: Link's overall framework. A fine-tuned NER model is used for recognizing entities and detecting uncertain entities, and the LLM reclassifies the detected uncertain entities.
• Figure 3: Illustration of the LinkNER framework and components.
• Figure 4: Illustration of local model (E-NER as an example) performance under different uncertainties. The polyline and histogram represent the changes in F1 score and entity density under different uncertainty intervals, respectively.
• Figure 5: Uncertainty threshold $\tau$ selection on the CoNLL'03 validation set. Align the fully linked LinkNER ($\tau=0.0$) prediction results with the results of the local model, and the intersection point is used as the threshold point.