Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement

TL;DR

The paper introduces the Judge's Verdict Benchmark to assess LLMs as judges for response accuracy by moving beyond correlation to agreement-based evaluation. It employs a two-step framework: first filtering judges with a strong linear relation to human judgments via Pearson $r$, then applying Cohen's Kappa $\kappa$ with a dynamic, mixed human group and a $z$-score test to classify judges as human-like ($|z|<1$) or super-consistent ($z>1$). Across 1,994 samples from six datasets, 27 of 54 LLMs reach Tier 1 excellence, split into 23 human-like and 4 super-consistent judges, with model size not being the sole determinant of performance. The work provides a rigorous, open benchmark and resources (dataset, code, leaderboards) to guide the design and selection of LLM-based evaluators for different evaluation goals and demonstrates the value of agreement-based validation for practical QA-with-ground-truth settings.

Abstract

This research introduces the Judge's Verdict Benchmark, a novel two-step methodology to evaluate Large Language Models (LLMs) as judges for response accuracy evaluation tasks. We assess how well 54 LLMs can replicate human judgment when scoring responses from RAG (Retrieval-Augmented Generation) or Agentic pipelines against ground truth answers. Our methodology progresses from traditional correlation analysis to comprehensive Cohen's Kappa analysis that measures actual agreement patterns. The two-step approach includes: (1) a correlation test that filters judges with strong alignment, followed by (2) a human-likeness test using z-scores to identify two distinct judgment patterns: human-like judgment (|z| < 1) that mimics natural human variation, and super-consistent judgment (z > 1) that exceeds typical human-to-human agreement levels. This methodology reveals that 27 out of 54 tested LLMs achieve Tier 1 performance: 23 models exhibit human-like patterns that preserve the nuances of human judgment, while 4 models demonstrate super-consistent behavior, a pattern that could indicate either enhanced reliability or oversimplification of complex judgments. Testing 43 open-source models (1B-405B parameters) and 11 closed models (GPT, Gemini, Claude variants), we demonstrate that judge excellence is not solely dependent on model size but on specific training strategies. Our key contributions include: (1) establishing that correlation alone is insufficient for judge evaluation, (2) introducing a "Turing Test for judges" based on agreement patterns, and (3) providing a standardized benchmark for classifying LLM judges into distinct performance tiers for different evaluation needs.

Figures (1)

• Figure 1: LLM Judge Evaluation Journey