Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge

TL;DR

The paper investigates position bias in rubric-based LLM-as-a-Judge, showing that LLMs exhibit systematic preferences for rubric options at certain positions. It introduces a balanced permutation strategy that evenly distributes each score across positions, enabling bias detection and calibration without changing the scoring semantics. Empirical results demonstrate that aggregating over balanced permutations can improve alignment with human judgments on HANNA and SummEval for several models, though effects vary by model and dataset. The method offers a simple, model-agnostic approach to improve rubric-based evaluation reliability and informs rubric ordering practices for robust automated judgments.

Abstract

Large language models (LLMs) are now widely used to evaluate the quality of text, a field commonly referred to as LLM-as-a-judge. While prior works mainly focus on point-wise and pair-wise evaluation paradigms. Rubric-based evaluation, where LLMs select a score from multiple rubrics, has received less analysis. In this work, we show that rubric-based evaluation implicitly resembles a multi-choice setting and therefore has position bias: LLMs prefer score options appearing at specific positions in the rubric list. Through controlled experiments across multiple models and datasets, we demonstrate consistent position bias. To mitigate this bias, we propose a balanced permutation strategy that evenly distributes each score option across positions. We show that aggregating scores across balanced permutations not only reveals latent position bias, but also improves correlation between the LLM-as-a-Judge and human. Our results suggest that rubric-based LLM-as-a-Judge is not inherently point-wise and that simple permutation-based calibration can substantially improve its reliability.

Figures (5)

• Figure 1: Three paradigms of LLM-as-a-Judge evaluation. Point-wise evaluation assigns a score given a question and a single response. Pair-wise evaluation compares two responses and outputs the model’s preference. Rubric-based evaluation further incorporates explicit scoring criteria. Judge models may also exhibit position bias toward responses in certain orders.
• Figure 2: Balanced permutation of rubric orderings. Aggregating the model’s choice distributions across permutations marginalizes out score identities and reveals systematic position bias.
• Figure 3: Unified prompt format used in all experiments (Prometheus-Eval).
• Figure 4: An illustration of balanced permutations of rubric scores, ensuring that each score appears equally often at each position.
• Figure 5: Position selection distribution on 4 datasets. The dashed red line indicates the expected 20% baseline under no position bias.