Label Curation Using Agentic AI
Subhodeep Ghosh, Bayan Divaaniaazar, Md Ishat-E-Rabban, Spencer Clarke, Senjuti Basu Roy
TL;DR
AURA addresses scalable, reliable data annotation for multi-modal datasets by coordinating heterogeneous off-the-shelf AI annotators without fine-tuning. It jointly infers latent true labels $y_i^\\star$ and annotator reliability via an EM-based adaptation of the Dawid–Skene model, computing posterior weights $w_{iy^\\star}$ and updating confusion matrices $\Theta^{(a)}$. Empirically, AURA outperforms individual annotators and majority voting across four datasets, with reliability estimates closely matching actual accuracies and robust behavior under class imbalance. This approach enables ground-truth-free, uncertainty-aware aggregation suitable for real-world annotation pipelines and heterogeneous model ecosystems.
Abstract
Data annotation is essential for supervised learning, yet producing accurate, unbiased, and scalable labels remains challenging as datasets grow in size and modality. Traditional human-centric pipelines are costly, slow, and prone to annotator variability, motivating reliability-aware automated annotation. We present AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation), an agentic AI framework for large-scale, multi-modal data annotation. AURA coordinates multiple AI agents to generate and validate labels without requiring ground truth. At its core, AURA adapts a classical probabilistic model that jointly infers latent true labels and annotator reliability via confusion matrices, using Expectation-Maximization to reconcile conflicting annotations and aggregate noisy predictions. Across the four benchmark datasets evaluated, AURA achieves accuracy improvements of up to 5.8% over baseline. In more challenging settings with poor quality annotators, the improvement is up to 50% over baseline. AURA also accurately estimates the reliability of annotators, allowing assessment of annotator quality even without any pre-validation steps.