Training and Evaluating with Human Label Variation: An Empirical Study

TL;DR

This work tackles the challenge of human label variation by introducing differentiable soft metrics based on fuzzy-set theory to evaluate and train models when multiple valid annotations exist. It conducts a large empirical study across 6 HLV datasets, 14 training methods, and 6 evaluation metrics using two pretrained models, revealing that simple training on disaggregated annotations or soft labels often outperforms more complex, objective-specific methods. The authors show that soft micro F1 and PO-JSD are among the strongest evaluation metrics for HLV data and provide a theoretical relationship clarifying why soft accuracy is bounded by PO-JSD. They also perform an empirical meta-evaluation of metrics to guide metric choice, recommending reporting both soft micro F1 and PO-JSD for interpretable and robust evaluation in HLV contexts.

Abstract

Human label variation (HLV) challenges the standard assumption that a labelled instance has a single ground truth, instead embracing the natural variation in human annotation to train and evaluate models. While various training methods and metrics for HLV have been proposed, it is still unclear which methods and metrics perform best in what settings. We propose new evaluation metrics for HLV leveraging fuzzy set theory. Since these new proposed metrics are differentiable, we then in turn experiment with employing these metrics as training objectives. We conduct an extensive study over 6 HLV datasets testing 14 training methods and 6 evaluation metrics. We find that training on either disaggregated annotations or soft labels performs best across metrics, outperforming training using the proposed training objectives with differentiable metrics. We also show that our proposed soft micro F1 score is one of the best metrics for HLV data.

Figures (7)

• Figure 1: Soft accuracy and PO-JSD graphs for a binary classification problem on a single example where the true and the predicted judgement for the positive class is 0.5 and $q$ respectively.
• Figure 2: Performance difference (delta) with mean MV performance of TwHIN-BERT on ArMIS. The methods are sorted by their mean ranking across datasets, models, and metrics.
• Figure 3: Performance difference (delta) with mean MV performance on HS-Brexit and MD-Agreement. SmF1 with TwHIN-BERT predicts zero for all test instances on HS-Brexit so its entropy correlation is undefined. The methods are sorted by their mean ranking across datasets, models, and metrics.
• Figure 4: Performance difference (delta) with mean MV performance on ChaosNLI (both the SNLI and the MNLI portions), MFRC, and TAG datasets. ChaosNLI doesn't have annotator identity information, so AE, AEh, AR, ARh are inapplicable. AL is omitted from MFRC and TAG because it is developed for single-label rather than multilabel tasks. AEh with RoBERTa predicts zero for all test instances on several classes in TAG, so its entropy correlation is undefined. The methods are sorted by their mean ranking across datasets, models, and metrics.
• Figure 5: Relationships between method performance as given by the evaluation metrics (Value) and method scores produced by human judgements (Score) on the TAG dataset. The shaded area denotes a 95% confidence interval. Each data point corresponds to a specific training method (e.g., ReL). AEh is excluded from RoBERTa with entropy correlation because of its degenerate results.

Theorems & Definitions (2)

• theorem 1
• proof