Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crowdsourcing with Difficulty: A Bayesian Rating Model for Heterogeneous Items

Seong Woo Han, Ozan Adıgüzel, Bob Carpenter

TL;DR

This study introduces a general purpose measurement-error model with which it can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability, and validate its goodness of fit with posterior predictive checks, the Bayesian analogue of $\chi^2$ tests.

Abstract

In applied statistics and machine learning, the "gold standards" used for training are often biased and almost always noisy. Dawid and Skene's justifiably popular crowdsourcing model adjusts for rater (coder, annotator) sensitivity and specificity, but fails to capture distributional properties of rating data gathered for training, which in turn biases training. In this study, we introduce a general purpose measurement-error model with which we can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability. We further show how to constrain the bimodal posterior of these models to avoid (or if necessary, allow) adversarial raters. We validate our model's goodness of fit with posterior predictive checks, the Bayesian analogue of $χ^2$ tests. Dawid and Skene's model is rejected by goodness of fit tests, whereas our new model, which adjusts for item heterogeneity, is not rejected. We illustrate our new model with two well-studied data sets, binary rating data for caries in dental X-rays and implication in natural language.

Crowdsourcing with Difficulty: A Bayesian Rating Model for Heterogeneous Items

TL;DR

This study introduces a general purpose measurement-error model with which it can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability, and validate its goodness of fit with posterior predictive checks, the Bayesian analogue of tests.

Abstract

In applied statistics and machine learning, the "gold standards" used for training are often biased and almost always noisy. Dawid and Skene's justifiably popular crowdsourcing model adjusts for rater (coder, annotator) sensitivity and specificity, but fails to capture distributional properties of rating data gathered for training, which in turn biases training. In this study, we introduce a general purpose measurement-error model with which we can infer consensus categories by adding item-level effects for difficulty, discriminativeness, and guessability. We further show how to constrain the bimodal posterior of these models to avoid (or if necessary, allow) adversarial raters. We validate our model's goodness of fit with posterior predictive checks, the Bayesian analogue of tests. Dawid and Skene's model is rejected by goodness of fit tests, whereas our new model, which adjusts for item heterogeneity, is not rejected. We illustrate our new model with two well-studied data sets, binary rating data for caries in dental X-rays and implication in natural language.
Paper Structure (26 sections, 14 equations, 3 figures, 2 tables)

This paper contains 26 sections, 14 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Posterior predictive checks
  3. Leave-one-out cross-validation
  4. Previous work
  5. Contributions
  6. A general crowdsourcing model
  7. The rating data
  8. The generative model
  9. Generating categories.
  10. Generating ratings.
  11. Parameter priors
  12. Spammy and adversarial raters.
  13. Including covariate information
  14. Marginal likelihood
  15. Model reductions
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Graphical sketch of the IRT-3PL model. Sizes: $J$ number of annotators, $K =2$ number of categories, $I$ number of items, $N$ number of categories. Observed data: $rating, rater, item$ labels. Parameters: $\theta$ annotator accuracies/biases, $\pi$ category prevalence, $z$ true item category, $\beta$ item difficulty, $\delta$ item discrimination, $\lambda$ item guessing.
  • Figure 2: L2 norm of training error. The L2 norm of parameter estimation error, $||\widehat{\theta} - \theta||_2$, for different approaches to training with probabilities: (log odds) training a linear regression on the log odds, (max prob) assigning the highest probability category, (noisy odds) add standard normal noise to the log odds approach, (random) assign a random category according to the probability, (weighted) train a weighted logistic regression. The errors are consistent whether using Bayesian posterior means with a normal prior (left) or ridge-penalized maximum likelihood estimates (right). Regression is 32-dimensional, with correlated inputs and 1024 training data points. Results show standard bar-and-whisker plots over 32 trials with paired random $x, \beta$.
  • Figure 3: Distribution of positive votes per item comparing the baseline Dawid & Skene model (ABC) with IRT models with difficulty (AB), discrimination (A), and guessing (full) alongside actual data, demonstrating the varying levels of dispersion captured by each model.