Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

TL;DR

JudgeBiasBench is proposed, a benchmark for systematically quantifying biases in LLM-based judges, and bias-aware training that explicitly incorporates bias-related attributes into the training process, encouraging judges to disentangle task-relevant quality from bias-correlated cues.

Abstract

Large language model (LLM)-based judges are widely adopted for automated evaluation and reward modeling, yet their judgments are often affected by judgment biases. Accurately evaluating these biases is essential for ensuring the reliability of LLM-based judges. However, existing studies typically investigate limited biases under a single judge formulation, either generative or discriminative, lacking a comprehensive evaluation. To bridge this gap, we propose JudgeBiasBench, a benchmark for systematically quantifying biases in LLM-based judges. JudgeBiasBench defines a taxonomy of judgment biases across 4 dimensions, and constructs bias-augmented evaluation instances through a controlled bias injection pipeline, covering 12 representative bias types. We conduct extensive experiments across both generative and discriminative judges, revealing that current judges exhibit significant and diverse bias patterns that often compromise the reliability of automated evaluation. To mitigate judgment bias, we propose bias-aware training that explicitly incorporates bias-related attributes into the training process, encouraging judges to disentangle task-relevant quality from bias-correlated cues. By adopting reinforcement learning for generative judges and contrastive learning for discriminative judges, our methods effectively reduce judgment biases while largely preserving general evaluation capability.

Figures (5)

• Figure 1: Two examples of judgment errors. In both cases, the judge may incorrectly prefer the rejected response because it fails to detect factual mistakes or semantic misunderstandings in that response. Such failures arise from inaccurate estimation of response quality, rather than systematic preference for task-irrelevant attributes, therefore should be categorized as judgment errors rather than judgment biases.
• Figure 2: Overview of the JudgeBiasBench construction pipeline. Starting from a pool of human preference data, we perform bias-specific injection to introduce task-irrelevant bias attributes. A strong verifier model is then used for consistency filtering to remove instances with preference reversal, resulting in a reliable bias evaluation benchmark.
• Figure 3: The illustration of our proposed framework. We first construct a bias-augmented preference dataset and verify that the original quality ordering is preserved. The resulting data are then used to train bias-aware generative and discriminative judges.
• Figure 4: Scaling effects of generative judges.
• Figure 5: Scaling effects of discriminative judges.