Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale

Weiyue Li, Minda Zhao, Weixuan Dong, Jiahui Cai, Yuze Wei, Michael Pocress, Yi Li, Wanyan Yuan, Xiaoyue Wang, Ruoyu Hou, Kaiyuan Lou, Wenqi Zeng, Yutong Yang, Yilun Du, Mengyu Wang

TL;DR

The paper investigates how the numeric grading scale affects LLMs acting as evaluators and their alignment with human judgments. Using a fully crossed design with 12 human annotators and 6 LLMs across six benchmarks and three scales, the authors quantify absolute agreement via ICC and calibration via nMAE. They find that the 0-5 scale yields the strongest human–LLM alignment overall, while inter-scale consistency for LLMs is benchmark-dependent and weaker for subjective tasks, revealing a 'reliability illusion' when aggregating across benchmarks. The results underscore the need for scale-aware evaluation design and per-task diagnostic reporting to ensure reliable, fair LLM-based judgments in practice.

Abstract

Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.

Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale

TL;DR

The paper investigates how the numeric grading scale affects LLMs acting as evaluators and their alignment with human judgments. Using a fully crossed design with 12 human annotators and 6 LLMs across six benchmarks and three scales, the authors quantify absolute agreement via ICC and calibration via nMAE. They find that the 0-5 scale yields the strongest human–LLM alignment overall, while inter-scale consistency for LLMs is benchmark-dependent and weaker for subjective tasks, revealing a 'reliability illusion' when aggregating across benchmarks. The results underscore the need for scale-aware evaluation design and per-task diagnostic reporting to ensure reliable, fair LLM-based judgments in practice.

Abstract

Large language models (LLMs) are increasingly used as automated evaluators, yet prior works demonstrate that these LLM judges often lack consistency in scoring when the prompt is altered. However, the effect of the grading scale itself remains underexplored. We study the LLM-as-a-judge problem by comparing two kinds of raters: humans and LLMs. We collect ratings from both groups on three scales and across six benchmarks that include objective, open-ended subjective, and mixed tasks. Using intraclass correlation coefficients (ICC) to measure absolute agreement, we find that LLM judgments are not perfectly consistent across scales on subjective benchmarks, and that the choice of scale substantially shifts human-LLM agreement, even when within-group panel reliability is high. Aggregated over tasks, the grading scale of 0-5 yields the strongest human-LLM alignment. We further demonstrate that pooled reliability can mask benchmark heterogeneity and reveal systematic subgroup differences in alignment across gender groups, strengthening the importance of scale design and sub-level diagnostics as essential components of LLM-as-a-judge protocols.
Paper Structure (38 sections, 5 equations, 2 figures, 17 tables)

This paper contains 38 sections, 5 equations, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs as automated evaluators.
  4. Grading consistency of LLM judges.
  5. Human-LLM agreement.
  6. Methodology
  7. Metrics
  8. Intraclass correlation coefficient.
  9. Normalized mean absolute error.
  10. Human and LLM Reliability
  11. Human reliability.
  12. LLM reliability.
  13. Human-LLM agreement.
  14. Experiments
  15. Data
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Overview of our experimental pipeline for assessing LLM-as-a-judge reliability. The framework consists of four main stages: (1) Data & Task Selection: collecting 5,497 items across six diverse datasets spanning objective and subjective judgments. (2) Inter-scale Consistency (LLM-only): evaluating six LLM judges across 0-5, 0-10, and 0-100 scales to construct consistency matrices. (3) Human-LLM Agreement: collecting fully crossed ratings from 12 human annotators (stratified by gender) and LLMs on 150 pooled items. (4) Reliability & Scale Analysis: employing two-way random-effects ANOVA to calculate Intraclass Correlation Coefficients ($\mathrm{ICC(A,1)}$ and $\mathrm{ICC(A,k)}$) for quantifying how scale design influences rater calibration.
  • Figure 2: ICC vs nMAE diagnostic on SummEval (subjective) and STS-B (objective).