ReIFE: Re-evaluating Instruction-Following Evaluation

TL;DR

A thorough meta-evaluation of instruction following is presented, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators and identifies the best-performing base LLMs and evaluation protocols with a high degree of robustness.

Abstract

The automatic evaluation of instruction following typically involves using large language models (LLMs) to assess response quality. However, there is a lack of comprehensive evaluation of these LLM-based evaluators across two dimensions: the base LLMs and the evaluation protocols. Therefore, we present a thorough meta-evaluation of instruction following, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators. Our evaluation allows us to identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. Moreover, our large-scale evaluation reveals: (1) Base LLM performance ranking remains largely consistent across evaluation protocols, with less capable LLMs showing greater improvement from protocol enhancements; (2) Robust evaluation of evaluation protocols requires many base LLMs with varying capability levels, as protocol effectiveness can depend on the base LLM used; (3) Evaluation results on different datasets are not always consistent, so a rigorous evaluation requires multiple datasets with distinctive features. We release our meta-evaluation suite ReIFE, which provides the codebase and evaluation result collection for more than 500 LLM-evaluator configurations, to support future research in instruction-following evaluation.

Figures (28)

• Figure 1: Overview of our large-scale meta-evaluation study of instruction-following evaluation. We evaluate the capabilities of 25 open-source base LLMs and 15 evaluation protocols, resulting in a total of 375 LLM-evaluators -- evaluation methods that perform the evaluations using the base LLMs by following the evaluation protocols.
• Figure 2: Correlation between the base LLMs' evaluation accuracy with the base protocol and their average accuracy across 15 protocols. The fitted regression line and the 95% confidence interval are displayed.
• Figure 3: Evaluation protocols' evaluation accuracy with the stronger and weaker base LLM groups, with a fitted regression line and a 95% confidence interval.
• Figure 4: Distribution of the correctness rate (Eq. \ref{['eq:correctness']}) of data examples in each dataset over different LLM-evaluators.
• Figure 5: Spearman's correlations between the performance ranking of LLM-evaluators on different datasets.