Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation

TL;DR

This paper tackles the cost-quality tension in ground-truth data annotation by introducing minority reports as a target for pruning. It develops a prescriptive framework that predicts, in real time, the likelihood that a given annotation task will disagree with the majority and then selectively prunes high-risk tasks before execution, using a mixed-effects logistic model with crop and worker random effects and a bathtub fatigue term. Empirical analyses on two CV datasets show that image ambiguity, worker variability, and fatigue explain minority reports, enabling substantial reductions in annotations (up to ~60%) with only modest losses in label quality, and the authors provide both simulation results and a theoretical no-free-lunch analysis to bound the tradeoffs. The work offers practical guidelines for annotation platforms to tailor repeats and thresholds to application needs (e.g., autonomous driving) and lays a foundation for integrating fatigue-aware, selective-pruning strategies into real-world data labeling pipelines.

Abstract

High-quality data annotation is an essential but laborious and costly aspect of developing machine learning-based software. We explore the inherent tradeoff between annotation accuracy and cost by detecting and removing minority reports -- instances where annotators provide incorrect responses -- that indicate unnecessary redundancy in task assignments. We propose an approach to prune potentially redundant annotation task assignments before they are executed by estimating the likelihood of an annotator disagreeing with the majority vote for a given task. Our approach is informed by an empirical analysis over computer vision datasets annotated by a professional data annotation platform, which reveals that the likelihood of a minority report event is dependent primarily on image ambiguity, worker variability, and worker fatigue. Simulations over these datasets show that we can reduce the number of annotations required by over 60% with a small compromise in label quality, saving approximately 6.6 days-equivalent of labor. Our approach provides annotation service platforms with a method to balance cost and dataset quality. Machine learning practitioners can tailor annotation accuracy levels according to specific application needs, thereby optimizing budget allocation while maintaining the data quality necessary for critical settings like autonomous driving technology.

Figures (13)

• Figure 1: Overview of our method. Our prescriptive pruning framework can slot into any annotation pipeline after tasks are assigned but before they are executed to reduce costs.
• Figure 2: A frame from ECPD with annotated bounding boxes. The green boxes correspond to crops for pedestrians and the blue boxes to crops for motorcycle riders.
• Figure 3: The number of tasks completed by each annotator for the two datasets.
• Figure 4: For both annotation datasets: (Left) the histogram of disagreement rates per worker; (Right) the histogram of disagreement rates per crop.
• Figure 5: The hourly disagreement rate on each day of the study. The solid black lines reflect the average disagreement rate over all active workers during that hour and the light blue lines reflect the disagreement rates for individual workers.

Theorems & Definitions (9)

• Theorem 1
• Theorem 2
• Corollary 1
• Example 1
• proof
• Lemma EC.1
• proof
• proof
• proof