HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants

TL;DR

HumanRankEval tackles the scalable evaluation of LMs as conversational assistants by introducing a large, human-authored, ranked QA dataset derived from StackExchange and StackOverflow and a log-likelihood-based metric that measures the LM's alignment with human preferences. The core idea is to compute LM answer log-likelihoods $ll$ for each candidate response and correlate the resulting LM rankings with human rankings using Pearson correlation, micro-averaged over 7K questions. The study demonstrates that HRE cleanly differentiates pretrained from instruction-tuned models across multiple sizes and correlates highly with human judgments, outperforming several knowledge-based and LM-as-a-judge baselines in reflecting conversational behavior. These findings suggest HRE as a practical, scalable tool to accelerate development of conversational LMs, while acknowledging limitations around composite preference, potential data contamination, and language coverage.

Abstract

Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM's distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE's efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.

Figures (13)

• Figure 1: Overview of HumanRankEval: given a question with multiple answers, we correlate human scores of each answer with the log-likelihoods of the LM. The unabridged answers can be found in Figure \ref{['fig:dialogue']}.
• Figure 2: HumanRankEval example from StackOverflow (Java topic).
• Figure 3: Average votes per answer/topic. Each answer has approximately double the votes of the next answer. More details can be found in Figure \ref{['fig:appendix_votes']} in the Appendix.
• Figure 4: HumanRankEval (per-topic) scores for Pythia LMs.
• Figure 5: HumanRankEval (avg) scores for Pythia LMs.