Improving ML Training Data with Gold-Standard Quality Metrics

TL;DR

This work tackles the challenge of unreliable hand-tagged training data by employing inter-rater agreement metrics, notably Krippendorff's $\alpha$, across multiple tagging iterations to separate tagging noise from data ambiguity. It proposes a methodology combining task design, tagger education, and ongoing monitoring to achieve high-quality labels without tagging every item multiple times. Key findings show that moving variance in $\alpha$ decreases as taggers gain experience and receive targeted feedback, indicating burn-in is not sufficient early on. The approach offers practical data-collection savings and extends to other tagging tasks, with implications for improving supervised ML trained on human annotations.

Abstract

Hand-tagged training data is essential to many machine learning tasks. However, training data quality control has received little attention in the literature, despite data quality varying considerably with the tagging exercise. We propose methods to evaluate and enhance the quality of hand-tagged training data using statistical approaches to measure tagging consistency and agreement. We show that agreement metrics give more reliable results if recorded over multiple iterations of tagging, where declining variance in such recordings is an indicator of increasing data quality. We also show one way a tagging project can collect high-quality training data without requiring multiple tags for every work item, and that a tagger burn-in period may not be sufficient for minimizing tagger errors.

Figures (5)

• Figure 1: Krippendorff's Alpha
• Figure 2: Tagging Task Design Process
• Figure 3: Krippendorff's Alpha of Tagger Education Data
• Figure 4: Krippendorff's Alpha of Monitoring Datasets
• Figure 5: Moving Variance of Krippendorff's Alpha of Five Monitoring Datasets