The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs

TL;DR

This work introduces the Alternative Annotator Test (alt-test), a principled statistical procedure to justify replacing human annotators with LLMs by comparing LLM alignments to a group of humans on a modest subset of data. It pairs a leave-one-annotator-out framework with a cost-benefit parameter ε and a paired t-test, controlling for multiple comparisons via Benjamini-Yekutieli to derive a winning rate that supports replacement when the LLM outperforms a majority of humans. The authors also define Average Advantage Probability (ρ) to compare LLM judges, and prove an optimal-jury result: under ACC and -RMSE scoring, an LLM-as-a-judge can emulate the majority or mean of human annotations, achieving ρ = 1. Across ten diverse datasets and multiple LLMs and prompting strategies, the alt-test demonstrates that LLMs can, in many cases, match or surpass human annotators, especially with few-shot prompting, while highlighting dataset- and task-specific limitations. The work provides guidelines, extensions for imbalanced or subjective labels, and a framework to promote rigorous, transparent use of LLM annotations in NLP and other applied fields.

Abstract

The "LLM-as-an-annotator" and "LLM-as-a-judge" paradigms employ Large Language Models (LLMs) as annotators, judges, and evaluators in tasks traditionally performed by humans. LLM annotations are widely used, not only in NLP research but also in fields like medicine, psychology, and social science. Despite their role in shaping study results and insights, there is no standard or rigorous procedure to determine whether LLMs can replace human annotators. In this paper, we propose a novel statistical procedure, the Alternative Annotator Test (alt-test), that requires only a modest subset of annotated examples to justify using LLM annotations. Additionally, we introduce a versatile and interpretable measure for comparing LLM annotators and judges. To demonstrate our procedure, we curated a diverse collection of ten datasets, consisting of language and vision-language tasks, and conducted experiments with six LLMs and four prompting techniques. Our results show that LLMs can sometimes replace humans with closed-source LLMs (such as GPT-4o), outperforming the open-source LLMs we examine, and that prompting techniques yield judges of varying quality. We hope this study encourages more rigorous and reliable practices.

Figures (5)

• Figure 1: An Illustration of the Alt-Test: Given instances annotated by human annotators, we first exclude each annotator in turn to estimate the probabilities that the LLM better represents the remaining annotators and that the excluded annotator better represents them. We then test whether the LLM probability exceeds the annotator probability (considering a cost-benefit penalty $\varepsilon$), and apply a False Discovery Rate (FDR) controlling procedure. Then, we calculate the winning rate, $\omega$, as the proportion of rejected hypotheses. If $\omega \geq 0.5$, we conclude that the LLM is more likely to hold an advantage over human annotators, which justifies using it.
• Figure 2: Analysis of the Impact of the Number of Items: Each data point is calculated using a bootstrap of 100 combinations of three annotators and $n$ items (x-axis). The y-axis shows the winning rates ($\omega$, solid lines) for $\varepsilon=0.1$ (purple) and $\varepsilon=0.2$ (turquoise). In addition, it presents the average advantage probability ($\rho$, dashed brown line) with its empirical 0.9 confidence intervals. The subplot title indicates the examined LLM.
• Figure 3: Analysis of the Impact of Different $\varepsilon$ Values: The x-axis represents different $\varepsilon$ values, while the y-axis shows the winning rate $\omega$ for four LLMs. If $\omega \ge 0.5$ (red line with triangles), the LLM passes the test, indicating it is a comparable alternative to human annotators when considering the cost-benefit tradeoff represented by $\varepsilon$. The annotator types are stated next to the dataset names.
• Figure 4: Simulation-Based Analysis of Annotator and LLM Noise Dynamics: Each data point is calculated using a bootstrap of 2500 combinations of different gold label priors, three annotators, $n$ items (x-axis), and $K=4$ categories. The y-axis shows the winning rates ($\omega$, solid lines) for four $\varepsilon$ values. In addition, it presents the average advantage probability ($\rho$, dashed brown line) with its empirical 0.9 confidence intervals. The subplot titles indicate the noise levels: $\eta_h$ increases from left to right, and $\eta_f$ increases from top to bottom. Each subplot also reports the IAA Cohen’s $\kappa$ for the human annotators and the accuracy of the LLM with the majority vote.
• Figure 5: Simulation-Based Analysis of the Number of Categories: Please see the caption of Figure \ref{['fig:noise_simulation']}. We set $\eta_f=0.2$. The subplot titles indicate the human noise $\eta_h$, which increases from left to right, and the number of categories $K$, which increases from top to bottom.

Theorems & Definitions (3)

• Theorem 1: Optimal LLM-as-a-Judge
• Theorem 1: Optimal LLM-as-a-Judge
• proof