Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who can we trust? LLM-as-a-jury for Comparative Assessment

Mengjie Qian, Guangzhi Sun, Mark J. F. Gales, Kate M. Knill

TL;DR

BT-sigma is proposed, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone, and shows that BT-sigma consistently outperforms averaging-based aggregation methods.

Abstract

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-as-a-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminator strongly correlates with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

Who can we trust? LLM-as-a-jury for Comparative Assessment

TL;DR

BT-sigma is proposed, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone, and shows that BT-sigma consistently outperforms averaging-based aggregation methods.

Abstract

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-as-a-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminator strongly correlates with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.
Paper Structure (23 sections, 11 equations, 9 figures, 5 tables)

This paper contains 23 sections, 11 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NLG Evaluation with LLMs
  4. LLM Comparative Assessments
  5. Comparative Assessment
  6. Bradley-Terry Models
  7. Calibration and Probability Consistency
  8. Reliability-Aware Aggregation
  9. Experimental Setup
  10. Datasets.
  11. Results
  12. Bradley-Terry Performance and Probability Inconsistency
  13. Reliability-Aware Aggregation with BT-$\sigma$
  14. Judge Reliability Analysis
  15. Conclusions
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Hard/Soft Bradley–Terry (BT) and BT-$\sigma$ in the LLM-as-a-jury setting. Hard BT uses binary comparisons ($p(i\succ j) \in {0,1}$), while soft BT uses probabilistic preferences. BT-$\sigma$ extends soft BT with judge-specific discriminators $\sigma_{k}$ to model judge reliability.
  • Figure 2: Cycle inconsistency rate on different LLMs, evaluated on SummEval (COH).
  • Figure 3: SRC barchart for different aggregation methods on SummEval and Topical-Chat (ALL aspects).
  • Figure 4: Scatter plot of $1/\sigma_k$ and LLM performance with Avg-Prob, measured by SRC, on SummEval.
  • Figure 5: Cycle inconsistency rate on different LLMs, evaluated on SummEval.
  • ...and 4 more figures