Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

Yujie Wang, Chao Huang, Liner Yang, Zhixuan Fang, Yaping Huang, Yang Liu, Jingsi Yu, Erhong Yang

TL;DR

The paper tackles the problem of reducing expert annotation cost for span-based sequence labeling (e.g., NER, OEI) in crowdsourcing by addressing interdependent label structures. It introduces a Combinatorial Multi-Armed Bandit (CMAB) based online worker selection framework with three reward modes (Expert Only, MV, Expert+MV) and a kappa-thresholded ground-truth strategy to balance accuracy and cost. To enable offline evaluation on imbalanced, small-scale data, it proposes a SES data augmentation method that shifts, expands, and shrinks annotation spans to simulate realistic annotation errors. Empirical results on CoNLL 2003 and Chinese OEI show large $F_1$ gains (up to $100.04 imes$ the expert baseline) and substantial cost savings (up to $65.97 ext{ }\%$), and results hold under a dataset-independent Bernoulli-based evaluation; the approach also integrates with RLHF.

Abstract

This paper introduces a novel crowdsourcing worker selection algorithm, enhancing annotation quality and reducing costs. Unlike previous studies targeting simpler tasks, this study contends with the complexities of label interdependencies in sequence labeling. The proposed algorithm utilizes a Combinatorial Multi-Armed Bandit (CMAB) approach for worker selection, and a cost-effective human feedback mechanism. The challenge of dealing with imbalanced and small-scale datasets, which hinders offline simulation of worker selection, is tackled using an innovative data augmentation method termed shifting, expanding, and shrinking (SES). Rigorous testing on CoNLL 2003 NER and Chinese OEI datasets showcased the algorithm's efficiency, with an increase in F1 score up to 100.04% of the expert-only baseline, alongside cost savings up to 65.97%. The paper also encompasses a dataset-independent test emulating annotation evaluation through a Bernoulli distribution, which still led to an impressive 97.56% F1 score of the expert baseline and 59.88% cost savings. Furthermore, our approach can be seamlessly integrated into Reinforcement Learning from Human Feedback (RLHF) systems, offering a cost-effective solution for obtaining human feedback.

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

TL;DR

The paper tackles the problem of reducing expert annotation cost for span-based sequence labeling (e.g., NER, OEI) in crowdsourcing by addressing interdependent label structures. It introduces a Combinatorial Multi-Armed Bandit (CMAB) based online worker selection framework with three reward modes (Expert Only, MV, Expert+MV) and a kappa-thresholded ground-truth strategy to balance accuracy and cost. To enable offline evaluation on imbalanced, small-scale data, it proposes a SES data augmentation method that shifts, expands, and shrinks annotation spans to simulate realistic annotation errors. Empirical results on CoNLL 2003 and Chinese OEI show large gains (up to the expert baseline) and substantial cost savings (up to ), and results hold under a dataset-independent Bernoulli-based evaluation; the approach also integrates with RLHF.

Abstract

This paper introduces a novel crowdsourcing worker selection algorithm, enhancing annotation quality and reducing costs. Unlike previous studies targeting simpler tasks, this study contends with the complexities of label interdependencies in sequence labeling. The proposed algorithm utilizes a Combinatorial Multi-Armed Bandit (CMAB) approach for worker selection, and a cost-effective human feedback mechanism. The challenge of dealing with imbalanced and small-scale datasets, which hinders offline simulation of worker selection, is tackled using an innovative data augmentation method termed shifting, expanding, and shrinking (SES). Rigorous testing on CoNLL 2003 NER and Chinese OEI datasets showcased the algorithm's efficiency, with an increase in F1 score up to 100.04% of the expert-only baseline, alongside cost savings up to 65.97%. The paper also encompasses a dataset-independent test emulating annotation evaluation through a Bernoulli distribution, which still led to an impressive 97.56% F1 score of the expert baseline and 59.88% cost savings. Furthermore, our approach can be seamlessly integrated into Reinforcement Learning from Human Feedback (RLHF) systems, offering a cost-effective solution for obtaining human feedback.
Paper Structure (33 sections, 1 theorem, 10 equations, 7 figures, 7 tables, 2 algorithms)

This paper contains 33 sections, 1 theorem, 10 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. Worker Selection Algorithm
  6. Expert Only
  7. Majority Voting (MV)
  8. Expert+MV
  9. Data Augmentation Method
  10. Shifting
  11. Expanding
  12. Shrinking
  13. Experiments
  14. Original Datasets
  15. CoNLL 2003
  16. ...and 18 more sections

Key Result

Lemma 1

(Chernoff Bound motwani1995randomized) Let $X_1, X_2, \cdots, X_N$ be independent random variables such that $x_l \le X_i \le x_h$ for all $i \in \{1,2, \cdots, N\}$. Let $X=\sum_{i=1}^{N}X_i$ and $\mu=\mathbb{E}(X)$. Given any $\delta>0$, we have the following result:

Figures (7)

  • Figure 1: Our online worker selection framework for crowdsourcing.
  • Figure 2: An example of the three methods to generate annotations. Chinese characters and corresponding English words with red backgrounds indicate annotation spans.
  • Figure 3: Cumulative regrets w.r.t time steps of all different worker selection methods.
  • Figure 4: $\textrm{F}_1$ scores of the produced annotations and usage of expert for annotation evaluations w.r.t the kappa threshold $\tau$ of the Exp.+MV method on Chinese OEI and CoNLL 2003 datasets.
  • Figure 5: A case in which the crowd worker annotates a span with correct length and polarity but incorrect position.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Lemma 1