Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?

TL;DR

Crowd-Calibrator introduces a soft-calibration framework that uses crowd disagreement to govern when a model should predict or abstain in subjective NLP tasks. By training a crowd estimator to approximate human label distributions and calibrating the base model via proximity measures like KL, JSD, or TVD (augmented with entropy), the approach aligns model confidence with human subjectivity. Empirical results on hate speech detection and natural language inference demonstrate competitive or superior selective prediction performance, including cross-dataset transfer, particularly when ample per-instance annotations are available. The work highlights the value of incorporating human disagreement into model decision-making and suggests directions for subjectivity-aware evaluation and data collection in NLP.

Abstract

Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote. This obfuscates annotator disagreement and the inherent uncertainty of the label. We argue that subjectivity should factor into model decisions and play a direct role via calibration under a selective prediction setting. Specifically, instead of calibrating confidence purely from the model's perspective, we calibrate models for subjective tasks based on crowd worker agreement. Our method, Crowd-Calibrator, models the distance between the distribution of crowd worker labels and the model's own distribution over labels to inform whether the model should abstain from a decision. On two highly subjective tasks, hate speech detection and natural language inference, our experiments show Crowd-Calibrator either outperforms or achieves competitive performance with existing selective prediction baselines. Our findings highlight the value of bringing human decision-making into model predictions.

Figures (9)

• Figure 1: Example showing the difference between a hard and a soft label.
• Figure 2: Results of training RoBERTa-large using hard (green, left) vs. soft labels (orange, right). Arrows indicate if a higher or lower value is better.
• Figure 3: Visualizations of the confidence distributions for both perfect agreement and disagreement samples separately in the test set. Figure (a) shows the distribution for models trained with $CE_{hard}$ and Figure (b) shows it for models trained with $JSD$. In Figures (c) - (d), we visualize the confidence distributions for four different scenarios: $\{\text{Perfect Agreement, Disagreement}\} \times \{\text{Correct, Incorrect}\}$.
• Figure 4: Crowd-Calibrator: our proposed soft calibration approach where we calibrate a model according to human subjectivity. We only let the model predict if there is high agreement and thus a clear(er) correct answer. For NLI we directly train our crowd estimator to predict the crowd distribution as we only have access to the label distribution and not individual annotators.
• Figure 5: Results on the HateXplain dataset for both RoBERTa-base and RoBERTa-large when training with hard and soft losses (x-axis). Each plot is a different metric.