Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge

TL;DR

The paper interrogates how human label uncertainty distorts the evaluation of automatic evaluation methods for generative models. It demonstrates that traditional correlation metrics can misleadingly favor machine labels when human judgments are noisy, and that this illusion persists until human agreement is high. To address this, the authors introduce a distribution-aware metric, binned Jensen-Shannon Divergence (JS_b), and perception charts that visualize human-machine alignment without assuming a single gold label. They validate their claims across multiple NLP tasks and LLM-based evaluators, and provide practical recommendations for stratified analyses, multi-metric reporting, and visualization to improve interpretability. The work offers a concrete framework and open-source tools to more robustly assess automatic evaluation methods in perception-heavy tasks.

Abstract

The effectiveness of automatic evaluation of generative models is typically measured by comparing the labels generated via automation with labels by humans using correlation metrics. However, metrics like Krippendorff's $α$ and Randolph's $κ$ were originally designed to measure the reliability of human labeling, thus make assumptions about typical human labeling behavior, and these assumptions may not be applicable to machine generated labels. In this paper, we show how *relying on a single aggregate correlation score* can obscure fundamental differences between human labels and those from automatic evaluation, including LLM-as-a-Judge. Specifically, we demonstrate that when the proportion of samples with variation or uncertainty in human assigned labels is relatively high, machine labels (generated by automatic evaluation methods) may superficially appear to have similar or better correlation with the human majority label compared to the human-to-human (HH) correlation. This can create the illusion that labels from automatic evaluation approximates the human majority label. However, as the proportion of samples with consistent human labels increases, the correlation between machine and human labels fall well below HH correlation. Based on these findings, we first propose stratifying data by human label uncertainty to provide a more robust analysis of automatic evaluation performance. Second, recognizing that uncertainty and variation are inherent in perception-based human evaluations, such as those involving attitudes or preferences, we introduce a new metric - binned Jensen-Shannon Divergence for perception for such scenarios to better measure the effectiveness of automatic evaluations. We present visualization techniques -- perception charts, to contextualize correlation measures appropriately. We have open-sourced at https://github.com/amazon-science/BeyondCorrelation.

Figures (11)

• Figure 1: Krippendorff's $\alpha$ (K-$\alpha$) on subsets of MNLI-mismatched with varying uncertainty, using 3 LLMs -- Sonnet, Mistral, and Llama. As proportion of samples with consistent human labels increases (towards the right of X-axis), HH correlations are significantly higher than $\text{H}^w\text{M}^w$. $\text{H}^w\text{H}'$ represents the correlation between the human majority and another human annotator ($\text{H}'$). $\text{H}^w\text{R}^w$ compares the human majority with a random machine evaluator, serving as baseline performance.
• Figure 2: Simulating the impact of uncertainty by comparing with an automatic random labeler (R). When the proportion of samples with uncertainty is higher, even a random labeler can appear to have better correlation with a majority ($\text{H}^w$) or median ($\overline{\text{H}}$) human label. As the proportion of samples with consistency increases, the weakness of the random labeler become evident.
• Figure 3: Ordinal HM perception comparison chart: Visualization of human perception vs. machine labels binned by human median rating (LJ Mistral on Coherence): Dial at the top shows the percentage of samples that fall into the bin. Middle scores (median 2-4) have higher human uncertainty, where only 60% (less than 2 out of 3) of the human labels follow the median. The extremes scores (1, 5) have less uncertainty. While Mistral seems to mimic human perception when the human median is 3, it is also biased towards the rating 3 when the human ratings are between 2-4.
• Figure 4: Pairwise preference HM perception chart: Human preference distribution vs. LJ GPT-4 for a given majority label per model pair. T is tie. Compares model pair (Left) Claude-v1 (A) vs. GPT3.5-turbo (B) (Right) Alpaca-13B (A) vs. GPT-3.5-turbo (B). Note: Using GPT-4 LJ has only one judgment per item in the results from zheng2023judging, hence $\text{M}^w$ and $\text{M}$ are the same.
• Figure 5: Impact of noise on ordinal dataset Summeval