Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient multi-prompt evaluation of LLMs

Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj, Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, Mikhail Yurochkin

TL;DR

It is proved that PromptEval consistently estimates the performance distribution and its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench Hard, and LMentry; for example, PromptEval can accurately estimate performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations.

Abstract

Most popular benchmarks for comparing LLMs rely on a limited set of prompt templates, which may not fully capture the LLMs' abilities and can affect the reproducibility of results on leaderboards. Many recent works empirically verify prompt sensitivity and advocate for changes in LLM evaluation. In this paper, we consider the problem of estimating the performance distribution across many prompt variants instead of finding a single prompt to evaluate with. We introduce PromptEval, a method for estimating performance across a large set of prompts borrowing strength across prompts and examples to produce accurate estimates under practical evaluation budgets. The resulting distribution can be used to obtain performance quantiles to construct various robust performance metrics (e.g., top 95% quantile or median). We prove that PromptEval consistently estimates the performance distribution and demonstrate its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench Hard, and LMentry; for example, PromptEval can accurately estimate performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations. Moreover, we show how PromptEval can be useful in LLM-as-a-judge and best prompt identification applications.

Efficient multi-prompt evaluation of LLMs

TL;DR

It is proved that PromptEval consistently estimates the performance distribution and its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench Hard, and LMentry; for example, PromptEval can accurately estimate performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations.

Abstract

Most popular benchmarks for comparing LLMs rely on a limited set of prompt templates, which may not fully capture the LLMs' abilities and can affect the reproducibility of results on leaderboards. Many recent works empirically verify prompt sensitivity and advocate for changes in LLM evaluation. In this paper, we consider the problem of estimating the performance distribution across many prompt variants instead of finding a single prompt to evaluate with. We introduce PromptEval, a method for estimating performance across a large set of prompts borrowing strength across prompts and examples to produce accurate estimates under practical evaluation budgets. The resulting distribution can be used to obtain performance quantiles to construct various robust performance metrics (e.g., top 95% quantile or median). We prove that PromptEval consistently estimates the performance distribution and demonstrate its efficacy empirically on three prominent LLM benchmarks: MMLU, BIG-bench Hard, and LMentry; for example, PromptEval can accurately estimate performance quantiles across 100 prompt templates on MMLU with a budget equivalent to two single-prompt evaluations. Moreover, we show how PromptEval can be useful in LLM-as-a-judge and best prompt identification applications.
Paper Structure (39 sections, 7 theorems, 26 equations, 19 figures, 2 tables, 3 algorithms)

This paper contains 39 sections, 7 theorems, 26 equations, 19 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem statement
  4. Performance distribution and quantiles estimation
  5. The correctness model
  6. Performance distribution and quantiles estimation using the correctness model
  7. Sampling $Y_\mathcal{E}$
  8. Theoretical guarantees
  9. Assessing multi-prompt evaluation strategies
  10. General assessment
  11. Data
  12. Methods and baselines
  13. Key results
  14. Further applications of PromptEval
  15. Estimating the distribution of scores for the LLM-as-a-judge framework
  16. ...and 24 more sections

Key Result

Theorem 4.4

Under conditions condit1, condit2, and condit3, it is true that and that where $W_1(F, \hat{F})$ is the Wasserstein 1-distance between the distributions $F$ and $\hat{F}$.

Figures (19)

  • Figure 1: Average estimation error for performance quantiles across 100 templates given a limited budget (in multiples of one-template MMLU evaluations).
  • Figure 2: Performance distribution estimation errors measured with Wasserstein-1 distance on three benchmarks.
  • Figure 3: Performance quantile estimation errors for varying quantiles (columns) and benchmarks (rows).
  • Figure 4: Estimating LLM-as-a-judge distribution of scores for 100 prompt variations given to the judge.
  • Figure 5: Best-prompt identification.
  • ...and 14 more figures

Theorems & Definitions (14)

  • Theorem 4.4
  • Theorem I.1
  • Lemma I.2
  • proof
  • Lemma I.3
  • proof
  • Lemma I.4
  • proof
  • Lemma I.5
  • proof
  • ...and 4 more