Uncertainty Quantification for Named Entity Recognition via Full-Sequence and Subsequence Conformal Prediction
Matthew Singer, Srijan Sengupta, Karl Pazdernik
TL;DR
This work tackles the lack of uncertainty quantification in Named Entity Recognition by introducing a general, conformal-prediction-based framework that produces prediction sets over full-sentence labelings with finite-sample coverage at level $1-\alpha$. By applying inductive conformal prediction to a CRF-based NER model, the authors generate calibrated, context-aware prediction sets that can be tailored to unconditional or class-conditional coverage, and they extend the approach to subsequence-level and integrated sentence-level predictions. Key innovations include three baseline non-conformity scores, stratification by language and length, and multiple non-conformity score hybrids (Naive, Conditional, and RAPS) to control set efficiency; these are complemented by subsequence conformal prediction and an integrated framework with Šidák correction to manage family-wise error. Empirical results on CoNLL++ and WikiNEuRal across multiple base models demonstrate valid coverage, improved calibration under stratification, and efficient prediction sets, highlighting practical benefits for multilingual and long-text NER. The framework thus provides a principled, scalable path toward uncertainty-aware NLP pipelines with robust downstream applicability.
Abstract
Named Entity Recognition (NER) serves as a foundational component in many natural language processing (NLP) pipelines. However, current NER models typically output a single predicted label sequence without any accompanying measure of uncertainty, leaving downstream applications vulnerable to cascading errors. In this paper, we introduce a general framework for adapting sequence-labeling-based NER models to produce uncertainty-aware prediction sets. These prediction sets are collections of full-sentence labelings that are guaranteed to contain the correct labeling with a user-specified confidence level. This approach serves a role analogous to confidence intervals in classical statistics by providing formal guarantees about the reliability of model predictions. Our method builds on conformal prediction, which offers finite-sample coverage guarantees under minimal assumptions. We design efficient nonconformity scoring functions to construct efficient, well-calibrated prediction sets that support both unconditional and class-conditional coverage. This framework accounts for heterogeneity across sentence length, language, entity type, and number of entities within a sentence. Empirical experiments on four NER models across three benchmark datasets demonstrate the broad applicability, validity, and efficiency of the proposed methods.