Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim

TL;DR

The paper tackles the reliability of LLM-based judges by introducing an Item Response Theory (IRT) framework, specifically the Graded Response Model (GRM), to separate latent evaluation signals from sample quality. It defines two complementary dimensions—intrinsic consistency (prompt-induced stability) and human alignment (concordance with human judgments)—and validates a two-phase diagnostic workflow: Phase 1 assesses intrinsic reliability via prompt perturbations, while Phase 2 evaluates alignment with humans using discrimination breadth and distributional similarity. The methodology is demonstrated across NLP and vision benchmarks using multiple judge models, revealing modality- and task-dependent patterns in reliability and alignment. The work provides practical diagnostics to guide when LLM-based evaluation is trustworthy and how to diagnose sources of unreliability, with implications for benchmarking and model development in multi-modal settings.

Abstract

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

TL;DR

The paper tackles the reliability of LLM-based judges by introducing an Item Response Theory (IRT) framework, specifically the Graded Response Model (GRM), to separate latent evaluation signals from sample quality. It defines two complementary dimensions—intrinsic consistency (prompt-induced stability) and human alignment (concordance with human judgments)—and validates a two-phase diagnostic workflow: Phase 1 assesses intrinsic reliability via prompt perturbations, while Phase 2 evaluates alignment with humans using discrimination breadth and distributional similarity. The methodology is demonstrated across NLP and vision benchmarks using multiple judge models, revealing modality- and task-dependent patterns in reliability and alignment. The work provides practical diagnostics to guide when LLM-based evaluation is trustworthy and how to diagnose sources of unreliability, with implications for benchmarking and model development in multi-modal settings.

Abstract

While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offering limited insight into whether LLM judges themselves function as stable and reliable measurement instruments. To address this limitation, we introduce a two-phase diagnostic framework for assessing reliability of LLM-as-a-Judge, grounded in Item Response Theory (IRT). The framework adopts Graded Response Model (GRM) of IRT and formalizes reliability along two complementary dimensions: (1) intrinsic consistency, defined as the stability of measurement behavior under prompt variations, and (2) human alignment, capturing correspondence with human quality assessments. We empirically examine diverse LLM judges with this framework, and show that leveraging IRT-GRM yields interpretable signals for diagnosing judgments systematically. These signals provide practical guidance for verifying reliablity of LLM-as-a-Judge and identifying potential causes of unreliability.
Paper Structure (68 sections, 6 equations, 7 figures, 7 tables)

This paper contains 68 sections, 6 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Intrinsic Consistency
  4. Human Alignment
  5. Methodology
  6. Graded Response Model for LLM Judges
  7. Why Should We Focus on Latent Quality ($\theta$)?
  8. Prompt Variation Design
  9. Reliability Metrics
  10. Phase 1: Intrinsic Consistency
  11. Prompt Consistency ($C_V$).
  12. Marginal Reliability ($\rho$).
  13. Diagnostic Interpretation.
  14. Phase 2: Human Alignment
  15. Discrimination Breadth Ratio ($\theta_{\text{ratio}}$).
  16. ...and 53 more sections

Figures (7)

  • Figure 1: Median latent quality ($\theta$) by human score for SummEval-Coherence (up) and VIEScore-MIE PQ (down). Each line represents a different judge model; the black line denotes human.
  • Figure 2: Median $\theta$ by human score for all SummEval metrics.
  • Figure 3: Median $\theta$ by human score for all TopicalChat metrics.
  • Figure 4: Median $\theta$ by human score for all HelpSteer-2 metrics.
  • Figure 5: Median $\theta$ by human score for VIEScore-CIG (SC and PQ).
  • ...and 2 more figures