VerifiNER: Verification-augmented NER via Knowledge-grounded Reasoning with Large Language Models

TL;DR

VerifiNER introduces a post-hoc verification framework that augments NER with knowledge-grounded reasoning to identify and revise errors without retraining. It performs span factuality verification by querying a KB to generate candidate spans, then uses knowledge to re-assign types, and finally applies contextual relevance verification through multi-path LLM reasoning with consistency voting. Across GENIA and BC5CDR biomedical datasets, VerifiNER yields consistent precision gains with modest recall trade-offs and demonstrates robustness to unseen distributions, label-shifted settings, and low-resource training. The approach is model-agnostic, leveraging external knowledge and LLM capabilities to improve faithfulness of predictions in knowledge-intensive domains. This work highlights the potential of verification-based strategies to enhance reliability in NER and suggests applicability to other domains beyond biomedicine.

Abstract

Recent approaches in domain-specific named entity recognition (NER), such as biomedical NER, have shown remarkable advances. However, they still lack of faithfulness, producing erroneous predictions. We assume that knowledge of entities can be useful in verifying the correctness of the predictions. Despite the usefulness of knowledge, resolving such errors with knowledge is nontrivial, since the knowledge itself does not directly indicate the ground-truth label. To this end, we propose VerifiNER, a post-hoc verification framework that identifies errors from existing NER methods using knowledge and revises them into more faithful predictions. Our framework leverages the reasoning abilities of large language models to adequately ground on knowledge and the contextual information in the verification process. We validate effectiveness of VerifiNER through extensive experiments on biomedical datasets. The results suggest that VerifiNER can successfully verify errors from existing models as a model-agnostic approach. Further analyses on out-of-domain and low-resource settings show the usefulness of VerifiNER on real-world applications.

Figures (9)

• Figure 1: Top: Erroneous prediction of ConNER model on GENIA dataset. Bottom: Human verification process of correcting errors with knowledge from a KB. From knowledge, it is evident that labeling RNA is incorrect.
• Figure 2: Ratio of error types (%) of NER models. Detailed statistics are reported in Table \ref{['tab:pie_stat']}.
• Figure 3: Distribution of token-level deviation between predicted and gold spans for each model. The x-axis represents the difference in token length between entities where the predicted span and gold span overlaps.
• Figure 4: Overview of VerifiNER framework. (1) Using entity prediction by existing models, we (a) extract candidate spans to retrieve knowledge from KB and verify factuality of span accordingly. Then (b) using retrieved knowledge, we verify factuality of type by generating knowledge-grounded evidence. (2) Lastly, we take consistency voting to select a candidate that is the most contextually relevant, with the help of the reasoning ability of LLMs.
• Figure 5: Example of generated evidence used for type factuality verification.