Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

TL;DR

This work interrogates reported self-preference biases in LLM evaluators, arguing that many observed effects arise from evaluator uncertainty rather than genuine narcissism. By introducing an Evaluator Quality Baseline that uses outcome-matched proxies and a per-example paired testing framework, the authors show an average reduction of 89–90% in self-preference signals across 37,448 evaluation instances, with remaining bias primarily in more objective tasks like MMLU. They decompose bias into illegitimate versus legitimate self-preference and analyze vote-entropy to understand confidence in judgments, finding that entropy patterns persist even when self-bias is mitigated. The results emphasize the importance of controlling evaluator quality and uncertainty in meta-evaluation research and provide a practical pathway to more reliable automatic evaluation workflows.

Abstract

Recent research has shown that large language models (LLM) favor own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which evaluation biases are explained by narcissism versus general experimental confounds, distorting measurements of self-preference bias. We discover a core methodological confound which could reduce measurement error by 89.6%. Specifically, LLM evaluators may deliver self-preferring verdicts when the judge responds to queries which they completed incorrectly themselves; this would be true regardless of whether one of their responses is their own. To decouple self-preference signals from noisy outputs on hard problems, we introduce an Evaluator Quality Baseline, which compares the probability that a judge incorrectly votes for itself against the probability that it votes for an incorrect response from another model. Evaluating this simple baseline on 37,448 queries, only 51% of initial findings retain statistical significance. Finally, we turn towards characterizing the entropy of "easy" versus "hard" evaluation votes from LLM judges. Our corrective baseline enables future research on self-preference by eliminating noisy data from potential solutions. More widely, this work contributes to the growing body of work on cataloging and isolating judge-bias effects.

Figures (15)

• Figure 1: (top) Self-preference biases can be decomposed into examples a judge completed correctly (LSP) and incorrectly (ILSP). (bottom) Since bias comes from false positives on incorrect examples, bias measurements should be made by taking the difference against a baseline to measure uncertainty artifacts. Often, differences between a judge evaluating itself and other models disappear against the baseline.
• Figure 2: Judge Task Accuracy versus Illegitimate Self-Preference (Sec. \ref{['sec:verif']}). With our proposed baseline, reported bias drops substantially.
• Figure 3: More results on Judge Task Accuracy versus original (light) and updated (full) self-preference experiments.
• Figure 4: Model-level winrate of judges versus weighted average winrate of selected proxies. Each point represents a judge model on a specific dataset. The strong correlation ($R^2=79\%$) validates our proxy selection method.
• Figure 5: Shannon Entropy on hard (ILSP) example distributions is strongly correlated, regardless of whether a model $J$ is judging itself (x axis) or a similar proxy $K$ (y axis) ($R^2=73\%$).