Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Judging with Confidence: Calibrating Autoraters to Preference Distributions

Zhuohang Li, Xiaowei Li, Chengyu Huang, Guowang Li, Katayoon Goshvadi, Bo Dai, Dale Schuurmans, Paul Zhou, Hamid Palangi, Yiwen Song, Palash Goyal, Murat Kantarcioglu, Bradley A. Malin, Yuan Xue

TL;DR

This paper tackles the reliability gap in LLM-based judging by arguing that autoraters must model the full distribution of human preferences rather than a single label. It introduces a probabilistic autorater framework and two distribution-matching finetuning methods: direct SFT with probabilistic labels and RL from sparse binary labels with proper scoring rewards. Empirical results show significant improvements in alignment (lower MSE), calibration (lower ECE), and reduced positional bias, while preserving objective-task performance; data efficiency favors RL with many sparse labels. The approach achieves strong out-of-distribution alignment with human judgments (e.g., PandaLM) and demonstrates robustness across subjective and objective tasks, offering a scalable path toward more reliable AI alignment systems.

Abstract

The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

Judging with Confidence: Calibrating Autoraters to Preference Distributions

TL;DR

This paper tackles the reliability gap in LLM-based judging by arguing that autoraters must model the full distribution of human preferences rather than a single label. It introduces a probabilistic autorater framework and two distribution-matching finetuning methods: direct SFT with probabilistic labels and RL from sparse binary labels with proper scoring rewards. Empirical results show significant improvements in alignment (lower MSE), calibration (lower ECE), and reduced positional bias, while preserving objective-task performance; data efficiency favors RL with many sparse labels. The approach achieves strong out-of-distribution alignment with human judgments (e.g., PandaLM) and demonstrates robustness across subjective and objective tasks, offering a scalable path toward more reliable AI alignment systems.

Abstract

The alignment of large language models (LLMs) with human values increasingly relies on using other LLMs as automated judges, or ``autoraters''. However, their reliability is limited by a foundational issue: they are trained on discrete preference labels, forcing a single ground truth onto tasks that are often subjective, ambiguous, or nuanced. We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population. In this paper, we propose a general framework for calibrating probabilistic autoraters to any given preference distribution. We formalize the problem and present two learning methods tailored to different data conditions: 1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a reinforcement learning approach for sparse, binary labels. Our empirical results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution, with improved calibration and significantly lower positional bias, all while preserving performance on objective tasks.

Paper Structure

This paper contains 56 sections, 2 theorems, 18 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. A Probabilistic Framework for Calibrating Autoraters
  3. Problem Formulation
  4. Discrete Autorater.
  5. Probabilistic Autorater.
  6. Benefits of Probabilistic Autoraters
  7. Informativeness.
  8. Alignment.
  9. Calibration.
  10. Finetuning Autoraters to Match the Preference Distribution
  11. Setting 1: Direct Supervised Finetuning with Probabilistic Labels.
  12. Setting 2: Reinforcement Learning from Binary Labels via Piecewise Proper Rewards.
  13. Consistency Analysis
  14. Experimental Setup
  15. Calibration Data.
  16. ...and 41 more sections

Key Result

Proposition 1

Assume the policy class can realize, for each ${\bm{x}}$, a deterministic numeric output $p_{\bm{\theta}}({\bm{x}})\in[0,1]$ with $s_{\bm{\theta}}({\bm{x}})=1$. Then any global maximizer of $J_R$ in equation eq:popobj satisfies: Moreover, for both rewards, any stochasticity in the reported numeric value or any non-zero density associated with unparsable outputs strictly reduces the expected rewar

Figures (3)

  • Figure 1: Overview of discrete vs. probabilistic autoraters. Left: Given a user query and two candidate responses, a discrete autorater returns a single preference (e.g., "B is better"), collapsing annotator variability. A probabilistic autorater predicts the full preference distribution and is finetuned via SFT/RL to match the target preference distribution. Right: Our finetuned autorater vs. zero-shot probabilistic conversions of discrete autoraters, including Verbalized Confidence (VC), Self-Consistency (SC), and Logits (Lo), evaluated using Gemma-2-9B on JudgeLM val set. Alignment error is measured by MSE, calibration error by ECE, and agreement by percentage.
  • Figure 2: Positional bias by method for Qwen-2.5-7B. Each horizontal box shows the distribution of Symmetry Deviation ($\Delta_{SD}$): $0$ is swap-symmetric, $-1$ indicates bias toward A, and $+1$ toward B. The black solid line marks the median, while the green dashed line marks the mean.
  • Figure 3: Result of RL finetuning with probabilistic labels.

Theorems & Definitions (3)

  • Proposition 1: Fisher Consistency of Brier and Log Rewards
  • Lemma 2: Unbiasedness and variance of the multi-annotator estimate
  • proof