Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimation

TL;DR

This paper tackles the problem of obtaining well-calibrated uncertainty estimates for LLM judges in production. It introduces a linear probe trained with a Brier score loss on the judge’s hidden states to produce calibrated verdict probabilities without additional model training or multi-sample generation. Empirical results show the probes significantly outperform verbalized-confidence and multi-generation baselines across dense and MoE models, with substantial computational savings and robust generalization to unseen tasks. The approach exhibits a conservative calibration, offering safer operation in safety-critical contexts at a modest cost to easy-task accuracy, and demonstrates strong potential for practical deployment in industry-scale LLM judging systems.

Abstract

As LLM-based judges become integral to industry applications, obtaining well-calibrated uncertainty estimates efficiently has become critical for production deployment. However, existing techniques, such as verbalized confidence and multi-generation methods, are often either poorly calibrated or computationally expensive. We introduce linear probes trained with a Brier score-based loss to provide calibrated uncertainty estimates from reasoning judges' hidden states, requiring no additional model training. We evaluate our approach on both objective tasks (reasoning, mathematics, factuality, coding) and subjective human preference judgments. Our results demonstrate that probes achieve superior calibration compared to existing methods with $\approx10$x computational savings, generalize robustly to unseen evaluation domains, and deliver higher accuracy on high-confidence predictions. However, probes produce conservative estimates that underperform on easier datasets but may benefit safety-critical deployments prioritizing low false-positive rates. Overall, our work demonstrates that interpretability-based uncertainty estimation provides a practical and scalable plug-and-play solution for LLM judges in production.

Figures (9)

• Figure 1: Calibration of models across model architectures, datasets, and uncertainty estimation methods as measured by the Kuiper metric. We compare four approaches across dense prompted judges (LLaMA 8B/70B, Qwen 32B), dense fine-tuned judges (J1 family), and MoE prompted judges (GPT-OSS 20B, LLaMA Scout). Our probe-based method outperforms baseline approaches across all architectures and training paradigms. Results averaged across all evaluation datasets. Lower values indicate better calibration.
• Figure 2: Reliability diagrams using 10 bins for Qwen 32B (Prompted Judge) and J1 LLAMA 70B (Finetuned Judge) on JudgeBench. We notice probes generally improve calibration. The color and percentage in each bar present the proportion of data samples in each bin. The verbalized method is generally overconfident, while multi-generation methods (consistency, majority) may be underconfident (Qwen) or overconfident (LLAMA) depending on model families. Similar trends are observed in finetuned and prompted variants of the judges, as shown in Appendix \ref{['app:reliability_diagrams']}.
• Figure 3: Accuracy of models at different confidence thresholds for Probe and Verbalized uncertainity estimation methods.
• Figure 4: Probes trained on the middle layers perform best. Probe performance by transformer layer. Probes perform better when trained on middle layers, with performance typically peaking around layers 16-64 depending on model size.
• Figure 5: Reliability plots for Dense Models