Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models

TL;DR

This paper argues that widely used probability-based evaluation methods for large language models do not align with how models are used in practice, where open-ended generation is the primary mode of interaction. It compares generation-based, label-based, and sequence-based predictions across MMLU, TruthfulQA, and Belebele benchmarks and shows substantial misalignment, including low agreement between methods and weak correlation with human preferences. The work introduces an agreement metric to quantify reliability and demonstrates that current MCQ proxies can mislead about model capabilities, especially under few-shot and multilingual conditions. The authors advocate for holistic evaluation frameworks that prioritize free-text generation quality, real user interactions, and human-aligned benchmarks to drive robust LLM progress and trustworthy deployment.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various applications, fundamentally reshaping the landscape of natural language processing (NLP) research. However, recent evaluation frameworks often rely on the output probabilities of LLMs for predictions, primarily due to computational constraints, diverging from real-world LLM usage scenarios. While widely employed, the efficacy of these probability-based evaluation strategies remains an open research question. This study aims to scrutinize the validity of such probability-based evaluation methods within the context of using LLMs for Multiple Choice Questions (MCQs), highlighting their inherent limitations. Our empirical investigation reveals that the prevalent probability-based evaluation method inadequately aligns with generation-based prediction. Furthermore, current evaluation frameworks typically assess LLMs through predictive tasks based on output probabilities rather than directly generating responses, owing to computational limitations. We illustrate that these probability-based approaches do not effectively correspond with generative predictions. The outcomes of our study can enhance the understanding of LLM evaluation methodologies and provide insights for future research in this domain.

Figures (5)

• Figure 1: An illustration of label-based, sequence-based and generation-based predictions for evaluating LLMs on NLP benchmarks.
• Figure 2: Differences in label and sequence accuracies compared to generation accuracies across datasets.
• Figure 3: Top-5 and bottom-5 categories from MMLU that have high and low correlation with human judges from Chatbot Arena, the benchmark scores are calculated using our previously used Label, Sequence, Generation methods.
• Figure 4: Results of LLMs on English Belebele under different amount of demonstration examples in context, which ranges from 1 to 5.
• Figure 5: Results of LLMs on Belebele under multilingual data including Amharic (amh_Ethi), Chinese (zho_Hans), Russian (rus_Cyrl), Swahili (swh_Latn) and Arabic (arb_Arab).