Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach

TL;DR

This work introduces QTL, a real-life DS-NER benchmark with a tiny validation set, and reveals that existing DS-NER methods struggle under practical tuning constraints. It proposes CuPUL, a token-level curriculum-based PU learning framework that orders training data from easy to hard using disagreements among multiple voters and a PU-risk objective, with optional self-training to boost performance. Across QTL and six benchmark datasets, CuPUL achieves state-of-the-art reliability in DS-NER without relying on large validation data, demonstrating robustness to real-world noise and label imperfections. The results underscore the value of token-level curriculum design and PU learning for distantly supervised NER, with practical implications for deploying DS-NER in professional domains.

Abstract

Distantly-Supervised Named Entity Recognition (DS-NER) uses knowledge bases or dictionaries for annotations, reducing manual efforts but rely on large human labeled validation set. In this paper, we introduce a real-life DS-NER dataset, QTL, where the training data is annotated using domain dictionaries and the test data is annotated by domain experts. This dataset has a small validation set, reflecting real-life scenarios. Existing DS-NER approaches fail when applied to QTL, which motivate us to re-examine existing DS-NER approaches. We found that many of them rely on large validation sets and some used test set for tuning inappropriately. To solve this issue, we proposed a new approach, token-level Curriculum-based Positive-Unlabeled Learning (CuPUL), which uses curriculum learning to order training samples from easy to hard. This method stabilizes training, making it robust and effective on small validation sets. CuPUL also addresses false negative issues using the Positive-Unlabeled learning paradigm, demonstrating improved performance in real-life applications.

Figures (6)

• Figure 1: Overview of CuPUL
• Figure 2: CuPUL Analysis: (a)(b) are the Difficulty Scores Distribution of Wikigold and Twitter, (c)(d) are the Token Level Positive Error Rate and Mean Difficulty Scores for Each Curriculum Stage on Wikigold and Twitter.
• Figure 3: Screenshot for online annotation tool TeamTat.
• Figure 4: F1 scores of CuPUL on test data of Wikigold trained with Distant Labels (red) and Ensembled Labels from voters (blue) after each curriculum training stage.
• Figure 5: Span Level Precision, Recall, and F1 scores of CuPUL with respect to Number of Voters $V$.