Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov, Jinghong Chen

TL;DR

A variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components is introduced and CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy.

Abstract

LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be eliminated by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy. It eliminates bias precisely while requiring each judge only once per cycle, maintaining the cost of single-judge evaluation. Empirical validation on MT-Bench supports all theoretical predictions.

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

TL;DR

A variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components is introduced and CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy.

Abstract

LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be eliminated by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy. It eliminates bias precisely while requiring each judge only once per cycle, maintaining the cost of single-judge evaluation. Empirical validation on MT-Bench supports all theoretical predictions.
Paper Structure (39 sections, 2 theorems, 35 equations, 2 figures, 2 tables)

This paper contains 39 sections, 2 theorems, 35 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM judge biases and multi-judge approaches.
  4. Budget-constrained evaluation.
  5. Benchmark reliability and generation variance.
  6. Methodology
  7. Model Specification
  8. Variance Decomposition
  9. Allocation Strategies
  10. Strategy A: All judges per generation.
  11. Strategy B: Random single judge per generation.
  12. Strategy C: CyclicJudge (round-robin).
  13. Dominance of CyclicJudge.
  14. Experiments
  15. Domain and Data
  16. ...and 24 more sections

Key Result

Proposition 1

Figures (2)

  • Figure 1: Estimated judge biases $\hat{\gamma}_\ell = \bar{X}_{\cdot\cdot\ell} - \bar{X}_{\cdots}$ for each judge--model pair.
  • Figure 2: Benchmark score variance $\operatorname{Var}(\bar{X})$ vs. per-scenario budget for three strategies: all judges (A), random single judge (B), and CyclicJudge (C).

Theorems & Definitions (3)

  • Proposition 1: Variance decomposition
  • Lemma 2: Finite population correction
  • proof