Enhancing LLM-Based Data Annotation with Error Decomposition
Zhen Xu, Vedant Khatri, Yijun Dai, Xiner Liu, Siyan Li, Xuanming Zhang, Renzhe Yu
TL;DR
The paper tackles the challenge of using LLMs for subjective data annotation where task inherent ambiguity undermines single alignment metrics. It introduces a diagnostic evaluation framework that decomposes LLM annotation errors by source (task inherent vs model driven) and type (boundary vs conceptual) and couples it with a lightweight human annotation test to estimate task ambiguity. A statistical decomposition using distances $δ_M$ and $δ_H$ and derived probabilities partitions observed errors into $P^{T}_{boundary}$, $P^{M}_{boundary}$, $P^{T}_{concept}$, and $P^{M}_{concept}$ through a two step process. Validation on four educational ordinal datasets shows that very high alignment is often unattainable due to task ambiguity, and provides a low cost diagnostic tool to assess task suitability for LLM annotation and guide targeted improvements for downstream validity.
Abstract
Large language models offer a scalable alternative to human coding for data annotation tasks, enabling the scale-up of research across data-intensive domains. While LLMs are already achieving near-human accuracy on objective annotation tasks, their performance on subjective annotation tasks, such as those involving psychological constructs, is less consistent and more prone to errors. Standard evaluation practices typically collapse all annotation errors into a single alignment metric, but this simplified approach may obscure different kinds of errors that affect final analytical conclusions in different ways. Here, we propose a diagnostic evaluation paradigm that incorporates a human-in-the-loop step to separate task-inherent ambiguity from model-driven inaccuracies and assess annotation quality in terms of their potential downstream impacts. We refine this paradigm on ordinal annotation tasks, which are common in subjective annotation. The refined paradigm includes: (1) a diagnostic taxonomy that categorizes LLM annotation errors along two dimensions: source (model-specific vs. task-inherent) and type (boundary ambiguity vs. conceptual misidentification); (2) a lightweight human annotation test to estimate task-inherent ambiguity from LLM annotations; and (3) a computational method to decompose observed LLM annotation errors following our taxonomy. We validate this paradigm on four educational annotation tasks, demonstrating both its conceptual validity and practical utility. Theoretically, our work provides empirical evidence for why excessively high alignment is unrealistic in specific annotation tasks and why single alignment metrics inadequately reflect the quality of LLM annotations. In practice, our paradigm can be a low-cost diagnostic tool that assesses the suitability of a given task for LLM annotation and provides actionable insights for further technical optimization.