CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges

TL;DR

CalibraEval addresses selection bias in LLMs-as-Judges by learning a label-free calibration function that maps observed prediction distributions to unbiased ones, formulated as an inference-time optimization with a non-parametric order-preserving algorithm (NOA). It enforces swap-invariance under changes in option order and tokens, and uses a gradient-based NOA procedure together with a PAVA-based monotone calibration to produce a robust debiasing function $g^*$. Across RewardBench, MTBench, and PreferenceBench, CalibraEval yields higher consistency (ICC/Kappa) and lower bias (Rstd) with improved accuracy compared to strong baselines like Pride, CC, DC, and DI, demonstrating robustness to prompt templates, ID tokens, and in-context learning settings. The work presents a scalable, transferable calibration framework that enhances the reliability of automated LLM-based evaluation of natural language generation, with practical implications for fair and stable AI assessment.

Abstract

The use of large language models (LLMs) as automated evaluation tools to assess the quality of generated natural language, known as LLMs-as-Judges, has demonstrated promising capabilities and is rapidly gaining widespread attention. However, when applied to pairwise comparisons of candidate responses, LLM-based evaluators often exhibit selection bias. Specifically, their judgments may become inconsistent when the option positions or ID tokens are swapped, compromising the effectiveness and fairness of the evaluation result. To address this challenge, we introduce CalibraEval, a novel label-free method for mitigating selection bias during inference. Specifically, CalibraEval reformulates debiasing as an optimization task aimed at adjusting observed prediction distributions to align with unbiased prediction distributions. To solve this optimization problem, we propose a non-parametric order-preserving algorithm (NOA). This algorithm leverages the partial order relationships between model prediction distributions, thereby eliminating the need for explicit labels and precise mathematical function modeling.Empirical evaluations of LLMs in multiple representative benchmarks demonstrate that CalibraEval effectively mitigates selection bias and improves performance compared to existing debiasing methods. This work marks a step toward building more robust and unbiased automated evaluation frameworks, paving the way for improved reliability in AI-driven assessments

Figures (6)

• Figure 1: Illustration of selection bias in LLMs-as-Judges. Selection bias manifests in two aspects: prefers a specific position or prefers a specific token.
• Figure 2: Four different types of combinations. $t_1/t_2$ represents the option IDs (A/B), while $o_1/o_2$ denotes the corresponding option contents. An unbiased evaluator consistently ranks the responses regardless of changes in option order (Swap Positions) or option ID tokens (Swap Tokens), ensuring fairness and consistency in the results.
• Figure 3: Performance comparison across different prompt templates.
• Figure 4: Performance comparison across different ID tokens.
• Figure 5: Performance comparison under in-context learning.