Towards DS-NER: Unveiling and Addressing Latent Noise in Distant Annotations

TL;DR

This paper tackles latent noise in distantly supervised NER by first analyzing how different distant annotation methods induce unlabeled-entity (UEP) and noisy-entity (NEP) errors via a noise-transition framework. It then proposes a decoupled two-stage approach, Unlabeled Entity Selection (UES) and Noisy Positive Elimination (NPE), implemented on a span-based NER setup with warm-up on reliable negatives, confident negatives sampling, and self-confidence–driven pruning, yielding a hyperparameter-free NEP handling. Empirical evaluation across eight real-world DS-NER datasets and multiple annotation sources shows consistent improvements over strong baselines, with ablations confirming the separate contributions of UES and NPE. The method demonstrates robust performance and generalization across domains and annotation styles, offering practical benefits for scalable, annotation-efficient NER in diverse settings.

Abstract

Distantly supervised named entity recognition (DS-NER) has emerged as a cheap and convenient alternative to traditional human annotation methods, enabling the automatic generation of training data by aligning text with external resources. Despite the many efforts in noise measurement methods, few works focus on the latent noise distribution between different distant annotation methods. In this work, we explore the effectiveness and robustness of DS-NER by two aspects: (1) distant annotation techniques, which encompasses both traditional rule-based methods and the innovative large language model supervision approach, and (2) noise assessment, for which we introduce a novel framework. This framework addresses the challenges by distinctly categorizing them into the unlabeled-entity problem (UEP) and the noisy-entity problem (NEP), subsequently providing specialized solutions for each. Our proposed method achieves significant improvements on eight real-world distant supervision datasets originating from three different data sources and involving four distinct annotation techniques, confirming its superiority over current state-of-the-art methods.

Figures (5)

• Figure 1: A noisy sample produced through distant supervision techniques, encompassing KB-Matching, Dict-Matching, and LLM-supervised, where "Dead Sea" emerges as an incomplete annotation, and the second mention of "Jordan" stands as an incorrect annotation. These issues respectively illustrate the Unlabeled-Entity Problem (UEP) and the Noisy-Entity Problem (NEP).
• Figure 2: The standard label noise transition matrix on three datasets (CoNLL03, Webpage, and BC5CDR) annotated by four distantly supervised methods, i.e., KB-Matching (KBM), Dict-Matching (DM), ChatGPT and UniversalNER (UniNER). The vertical axis represents observed labels $\tilde{y}$, and the horizontal axis represents true labels $y^*$.
• Figure 3: The overall framework of our method. Left: An example of our span-based setting, illustrating the unlabeled entity problem and the noisy entity problem. Right: The process begins by warming up the model with entity spans and reliable negative spans. Subsequently, the model is used to filter out noisy entities and identify confident negatives for further training.
• Figure 4: $FN_R, FN_P$ of our method during training on synthetic datasets.
• Figure 5: 1-shot prompt example for ChatGPT to annotate CoNLL03.