To Err Is Human; To Annotate, SILICON? Reducing Measurement Error in LLM Annotation
Xiang Cheng, Raveesh Mayya, João Sedoc
TL;DR
The paper addresses measurement error in LLM-based text annotation by decomposing it into four sources: guideline-induced, baseline-induced, prompt-induced, and model-induced errors. It introduces SILICON, a structured four-phase workflow with sequential optimization, to reduce these errors, and validates it across seven management tasks using both expert and crowdsourced baselines, various prompts, and multiple LLMs. Key contributions include a cost-effective multi-LLM labeling strategy guided by First-Second Distance confidence, regression-based statistical equivalence tests for robust reproducibility, and an open-source SILICON toolkit to implement the framework. The findings show that refining guidelines, using expert baselines, targeting prompt structure (system versus user role), and evaluating multiple models yield improved LLM–human agreement, with open-parameter models often achieving equivalence to proprietary models, supporting sustainable, reproducible annotation. The work provides practical guidelines for practitioners and demonstrates how to balance accuracy, cost, and long-term accessibility in LLM-driven annotation pipelines suitable for management research and beyond.
Abstract
Unstructured text data annotation is foundational to management research and Large Language Models (LLMs) promise a cost-effective and scalable alternative to human annotation. The validity of insights drawn from LLM annotated data critically depends on minimizing the discrepancy between LLM assigned labels and the unobserved ground truth, as well as ensuring long-term reproducibility of results. We address the gap in the literature on LLM annotation by decomposing measurement error in LLM-based text annotation into four distinct sources: (1) guideline-induced error from inconsistent annotation criteria, (2) baseline-induced error from unreliable human reference standards, (3) prompt-induced error from suboptimal meta-instruction formatting, and (4) model-induced error from architectural differences across LLMs. We develop the SILICON methodology to systematically reduce measurement error from LLM annotation in all four sources above. Empirical validation across seven management research cases shows iteratively refined guidelines substantially increases the LLM-human agreement compared to one-shot guidelines; expert-generated baselines exhibit higher inter-annotator agreement as well as are less prone to producing misleading LLM-human agreement estimates compared to crowdsourced baselines; placing content in the system prompt reduces prompt-induced error; and model performance varies substantially across tasks. To further reduce error, we introduce a cost-effective multi-LLM labeling method, where only low-confidence items receive additional labels from alternative models. Finally, in addressing closed source model retirement cycles, we introduce an intuitive regression-based methodology to establish robust reproducibility protocols. Our evidence indicates that reducing each error source is necessary, and that SILICON supports reproducible, rigorous annotation in management research.