Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge

TL;DR

This work tackles evaluating LLM-generated text with limited labelled data by framing LLM-as-a-judge within a generalized probabilistic model. It generalises Product-of-Experts (PoE) for comparing candidates, introduces generalized comparative experts and combines absolute with comparative scoring in a unified framework, including a learnable home-advantage term. Key contributions include Beta and Gaussian extensions for the pairwise likelihood, two targeted uncertainty measures (pairwise and ranking) with a Laplace-based posterior, and the probability-of-reordering criterion for efficient iterative selection, achieving notable efficiency gains. Empirical results on SummEval and HANNA show that uncertainty estimation largely drives efficiency improvements (roughly 50% fewer comparisons) and that ranking-level uncertainty helps identify low-performing predictions, while the exact choice of expert model has limited impact when uncertainty estimation is properly exploited.

Abstract

This paper explores generalised probabilistic modelling and uncertainty estimation in comparative LLM-as-a-judge frameworks. We show that existing Product-of-Experts methods are specific cases of a broader framework, enabling diverse modelling options. Furthermore, we propose improved uncertainty estimates for individual comparisons, enabling more efficient selection and achieving strong performance with fewer evaluations. We also introduce a method for estimating overall ranking uncertainty. Finally, we demonstrate that combining absolute and comparative scoring improves performance. Experiments show that the specific expert model has a limited impact on final rankings but our proposed uncertainty estimates, especially the probability of reordering, significantly improve the efficiency of systems reducing the number of needed comparisons by ~50%. Furthermore, ranking-level uncertainty metrics can be used to identify low-performing predictions, where the nature of the probabilistic model has a notable impact on the quality of the overall uncertainty.

Figures (9)

• Figure 1: The Spearman Rank Correlation when iteratively selecting the next examples of lowest confidence/highest uncertainty. The baseline is the soft Bradley-Terry model with the minimum uncertainty metric. We also report the proposed variance and probability of reordering under the soft BT model. Furthermore, the $\Phi$-Gaussian model is referred to as "ngaussian". Debiased refers to permutation debiased probabilities.
• Figure 2: Extended results on Flan-T5: (a) Comparing the soft BT model to the combined model which relies on additional absolute experts. Both use the probability of reordering as the selection criteria. (b) Selection performance. Varying the parameters $\alpha$ and $\beta$ of the Beta distribution in Eq. (\ref{['eq:gen beta']}). (c) Selection performance. Varying the parameter $\epsilon$ in Eq. (\ref{['eq:exp-sweep']}).
• Figure 3: The Spearman Rank Correlation when iteratively selecting the next batch $b$ of examples of lowest confidence/highest uncertainty. The baselines are Qwen2.5-{3B-7B} models with all comparisons selected. Furthermore, the efficient baseline is set to the soft BT model with the minimum uncertainty metric.
• Figure 4: The reliability diagram of biased and debiased, standard and calibrated systems on the coherency metric.
• Figure 5: The accuracy at a comparison-level when the examples of lowest confidence/highest uncertainty are rejected.