Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

TL;DR

The paper tackles the challenge of reliably ranking LLMs in alignment with human preferences by formalizing automatic benchers with four core components and evaluating three meta-evaluation settings. Through controlled experiments across input sets, evaluation models, evaluation types, and aggregation methods, it demonstrates that Arena Hard as the input set and base pairwise evaluation with Bradley-Terry or win ratio generally yield the strongest agreement with human judgments, while 5-point and pointwise approaches often underperform, especially for open-source evaluators. It also shows that benchers degrade when comparing closely matched systems and that instance-level evaluator rankings do not always translate to system-level bencher performance, underscoring the need for dedicated system-level evaluation. The work offers practical design recommendations and highlights limitations, providing code and data to support replication and further research in robust LLM alignment benchmarking.

Abstract

Evaluating and ranking the capabilities of different LLMs is crucial for understanding their performance and alignment with human preferences. Due to the high cost and time-consuming nature of human evaluations, an automatic LLM bencher (i.e., an automatic evaluation framework that aims to rank LLMs based on their alignment with human preferences) is indispensable. An automatic LLM bencher consists of four components: the input set (e.g., a user instruction), the evaluation model (e.g., an LLM), the evaluation type (e.g., pairwise comparison), and the aggregation method (e.g., the ELO rating system). However, previous work has not thoroughly explored how to select these components or how their different combinations influence the results. In this work, through controlled experiments, we provide a series of recommendations on how to choose each component to better automate the evaluation of LLMs. Furthermore, we discovered that when evaluating LLMs with similar performance, the performance of the automatic LLM bencher declines sharply, underscoring the limitations of current benchers and calling for future work. Lastly, we found that the evaluation models' performance at the instance level (e.g., the accuracy of selecting the best output) does not always align with their effectiveness when used as a component of a bencher, highlighting the importance of dedicated system-level evaluation of benchers.

Figures (16)

• Figure 1: Bootstrapping mean and 95% confidence interval of the correlations between the system rankings of automatic pairwise evaluation and human judgment when the total number of examples is set equal to that of pointwise evaluation. Results for pointwise and pairwise evaluations using gpt-4-0314 as the reference system are shown as horizontal lines. Both input sets and comparisons are sampled with replacement, with 1000 iterations. See Figure \ref{['fig:p_pairs_sampling2']} and \ref{['fig:p_pairs_sampling3']} in the appendix for other evaluation models.
• Figure 2: Controllable Kendall's tau ($\tau_u$) between the system rankings from automatic LLM benchers and human judgment when only partial system pairs are used. The X-axis denotes the value of threshold $u$, which controls the maximum difference between the systems' ChatBot Arena ratings. For pairwise_base_ref, we show the results with gpt-4-0314 as the reference system. Across all settings, we found that the benchers' performance degrades when they evaluate close-performing systems. See Figure \ref{['fig:controllable_kendall_tau_2']} and \ref{['fig:controllable_kendall_tau_3']} in the appendix for other evaluation models.
• Figure 3: Spearman's $\rho$ between the evaluation model rankings under different meta-evaluation settings. alpaca_eval and arena_hard denote the evaluation model rankings are produced under Setting 1. alpaca_farm_system denotes that evaluation model rankings are produced under Setting 3 where the instance-level human judgment of Alpaca Farm is used as ground truth, the Bradley-Terry model is used to aggregate both automatic evaluation and human evaluation. alpaca_farm_instance, mtbench, llmbar_natural, and llmbar_adversarial refer to that the evaluation model rankings are produced under Setting 2 where instance-level human judgment of Alpaca Farm, MT-Bench, LLMBAR are used.
• Figure 4: The relationships between the performance of the bencher and the Chatbot Arena ratings of the LLM used as the reference systems. The evaluation type is fixed as base pairwise.
• Figure 5: The relationships between the performance of the bencher and the Chatbot Arena ratings of the LLM used as the reference systems. The evaluation type is fixed as base pairwise.