Exploring the Effects of Alignment on Numerical Bias in Large Language Models

TL;DR

The paper addresses numerical bias in LLM evaluators (LLM-as-a-judge) and the role of alignment in amplifying this bias. It compares pre- and post-alignment models across MTQE, GECQE, and LCP, using kurtosis and Pearson correlation with human scores to quantify bias and accuracy. The study finds that alignment increases numerical bias and can reduce evaluation accuracy, though post-alignment models often outperform pre-alignment ones; among mitigation strategies, adjusting the score range is the most effective, with temperature scaling and distribution calibration offering partial gains. The work provides practical guidance on selecting evaluator LLMs, tuning prompts, and treating score-range as a tunable hyperparameter, while highlighting the need for robust, generalizable bias-mitigation approaches.

Abstract

"LLM-as-a-judge," which utilizes large language models (LLMs) as evaluators, has proven effective in many evaluation tasks. However, evaluator LLMs exhibit numerical bias, a phenomenon where certain evaluation scores are generated disproportionately often, leading reduced evaluation performance. This study investigates the cause of this bias. Given that most evaluator LLMs are aligned through instruction tuning and preference tuning, and that prior research suggests alignment reduces output diversity, we hypothesize that numerical bias arises from alignment. To test this, we compare outputs from pre- and post-alignment LLMs, and observe that alignment indeed increases numerical bias. We also explore mitigation strategies for post-alignment LLMs, including temperature scaling, distribution calibration, and score range adjustment. Among these, score range adjustment is most effective in reducing bias and improving performance, though still heuristic. Our findings highlight the need for further work on optimal score range selection and more robust mitigation strategies.

Figures (25)

• Figure 1: LLM evaluators with numerical biases tend to produce skewed scores regardless of input. In some cases, the results deviate from the distribution of manually assigned scores, questioning the robustness of the evaluation framework.
• Figure 2: English to German (En-De)
• Figure 3: English to Chinese (En-Zh)
• Figure 4: Nepali to English (Ne-En)
• Figure 5: Sinhala to English (Si-En)