Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization

Kohei Tsuji, Tatsuya Hiraoka, Yuchang Cheng, Tomoya Iwakura

TL;DR

SubRegWeigh tackles annotation errors in NLP by using subword regularization to implicitly generate multiple inference views from a single model, enabling fast annotation weighing. The method trains a scouting model on deterministic tokenization, infers with multiple tokenizations ($K$) at inference, and assigns weights to training samples based on agreement with original labels, using a weighted loss for the final model. Empirically, SubRegWeigh, especially with $K$-means selection of tokenizations, achieves 4–5× faster weighing than CrossWeigh and yields performance gains on CoNLL-CW (and competitive results on NER and SST-2) while effectively flagging pseudo-incorrect labels. The approach is task-agnostic beyond NER, scalable, and aligns with Green AI goals by reducing compute while improving annotation robustness.

Abstract

NLP datasets may still contain annotation errors, even when they are manually annotated. Researchers have attempted to develop methods to automatically reduce the adverse effect of errors in datasets. However, existing methods are time-consuming because they require many trained models to detect errors. This paper proposes a time-saving method that utilizes a tokenization technique called subword regularization to simulate multiple error detection models for detecting errors. Our proposed method, SubRegWeigh, can perform annotation weighting four to five times faster than the existing method. Additionally, SubRegWeigh improved performance in document classification and named entity recognition tasks. In experiments with pseudo-incorrect labels, SubRegWeigh clearly identifies pseudo-incorrect labels as annotation errors. Our code is available at https://github.com/4ldk/SubRegWeigh .

SubRegWeigh: Effective and Efficient Annotation Weighing with Subword Regularization

TL;DR

SubRegWeigh tackles annotation errors in NLP by using subword regularization to implicitly generate multiple inference views from a single model, enabling fast annotation weighing. The method trains a scouting model on deterministic tokenization, infers with multiple tokenizations () at inference, and assigns weights to training samples based on agreement with original labels, using a weighted loss for the final model. Empirically, SubRegWeigh, especially with -means selection of tokenizations, achieves 4–5× faster weighing than CrossWeigh and yields performance gains on CoNLL-CW (and competitive results on NER and SST-2) while effectively flagging pseudo-incorrect labels. The approach is task-agnostic beyond NER, scalable, and aligns with Green AI goals by reducing compute while improving annotation robustness.

Abstract

NLP datasets may still contain annotation errors, even when they are manually annotated. Researchers have attempted to develop methods to automatically reduce the adverse effect of errors in datasets. However, existing methods are time-consuming because they require many trained models to detect errors. This paper proposes a time-saving method that utilizes a tokenization technique called subword regularization to simulate multiple error detection models for detecting errors. Our proposed method, SubRegWeigh, can perform annotation weighting four to five times faster than the existing method. Additionally, SubRegWeigh improved performance in document classification and named entity recognition tasks. In experiments with pseudo-incorrect labels, SubRegWeigh clearly identifies pseudo-incorrect labels as annotation errors. Our code is available at https://github.com/4ldk/SubRegWeigh .
Paper Structure (38 sections, 2 equations, 2 figures, 10 tables)

This paper contains 38 sections, 2 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Annotation Error Detection
  4. Subword Regularization
  5. Proposed Method: SubRegWeigh
  6. Training of the Scouting Model
  7. Inference with the Scouting Model
  8. Random:
  9. Cos-Sim:
  10. $K$-means:
  11. Weighting for Training Samples
  12. Training of Final Model
  13. Experiments
  14. Dataset
  15. NER:
  16. ...and 23 more sections

Figures (2)

  • Figure 1: The overview of the three steps of SubRegWeigh with the number of tokenization candidates in the inference $K=3$ for the classification task whether the input sentence is positive or negative: 1) the training step of the scouting model, 2) the inference step to examine the appropriateness of the training label, and 3) the weighting step to calculate the weight of each training sample according to the appropriateness of their labels. Check marks indicate inferences that are the same as the original label and cross marks indicate inferences that differ from the original label. The calculated weights are used to train the final model.
  • Figure 2: The overview of SubRegWeigh in the case of NER.