Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
Kayla Schroeder, Zach Wood-Doughty
TL;DR
This paper tackles the problem of unreliable LLM judgments when used as evaluators by focusing on internal consistency rather than single-shot outputs. It introduces a reliability framework that uses McDonald's omega to quantify how consistently an LLM judge applies its evaluation criteria across replicated prompts while varying only the random seed. Through experiments on BBH, SQuAD, MT-Bench, and Head-to-Tail setups across multiple temperatures and models, the study reveals that reliability is not universal and can be significantly affected by seed variation and task type, with some models showing meaningful reliability gaps. The findings emphasize the need for seed-aware, multi-sample evaluation to build trustworthy LLM-based judgment systems and provide practical guidance for deploying reliable LLM evaluators in high-stakes settings.
Abstract
Large Language Models (LLMs) have become increasingly powerful and ubiquitous, but their stochastic nature poses challenges to the reliability of their outputs. While deterministic settings can improve consistency, they do not guarantee reliability, as a single sample from the model's probability distribution can still be misleading. Building upon the concept of LLM-as-a-judge, we introduce a novel framework for rigorously evaluating the reliability of LLM judgments, leveraging McDonald's omega. We evaluate the reliability of LLMs when judging the outputs of other LLMs on standard single-turn and multi-turn benchmarks, simultaneously investigating the impact of temperature on reliability. By analyzing these results, we demonstrate the limitations of fixed randomness and the importance of considering multiple samples, which we show has significant implications for downstream applications. Our findings highlight the need for a nuanced understanding of LLM reliability and the potential risks associated with over-reliance on single-shot evaluations. This work provides a crucial step towards building more trustworthy and reliable LLM-based systems and applications.