The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators

TL;DR

This work shows that pairwise evaluation of LLM outputs amplifies biased preferences, particularly under adversarial conditions. It analyzes two meta-evaluation datasets to reveal that pointwise evaluation is more robust to manipulation, then introduces PRePair, a hybrid method that performs pointwise reasoning for each candidate before a final pairwise decision. Across multiple models and benchmarks, PRePair significantly improves adversarial robustness (e.g., a $24.48\%$ average gain on the LLMBar-Adversarial set) while preserving or enhancing performance on standard MT-Bench tasks. The approach offers a practical path to more reliable LLM evaluators by combining the contextual strengths of pairwise assessment with the bias-resilience of pointwise reasoning.

Abstract

As large language models (LLMs) are increasingly used as evaluators for natural language generation tasks, ensuring unbiased assessments is essential. However, LLM evaluators often display biased preferences, such as favoring verbosity and authoritative tones. Our empirical analysis reveals that these biases are exacerbated in pairwise evaluation, where LLMs directly compare two outputs and easily prioritize superficial attributes. In contrast, pointwise evaluation, which assesses outputs independently, is less susceptible to such bias because each output is judged in isolation. To address the limitations of the pairwise evaluation, we introduce a novel evaluation method, PRePair, which integrates pointwise reasoning within a pairwise framework. PRePair effectively alleviates biased preference, improving performance on the adversarial benchmark (LLMBar) while outperforming pointwise evaluation on the standard benchmark (MT-Bench).

Figures (20)

• Figure 1: Results of pointwise and pairwise approaches in normal (MT-Bench) and adversarial (LLMBar-Adversarial) meta-evaluation datasets. The difference values represent the change between pointwise and pairwise approaches for each model under both normal and adversarial conditions.
• Figure 2: The overall illustration of PRePair.
• Figure 3: The strengths and weaknesses prompt for pairwise evaluation.
• Figure 4: The default prompt for pointwise evaluation with LLMBar dataset.
• Figure 5: The default prompt for pairwise evaluation with LLMBar dataset.