Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Evaluation of LLM Performance with Statistical Guarantees

Skyler Wu, Yash Nair, Emmanuel J. Candès

TL;DR

This work tackles the problem of reliably evaluating large language models under a fixed, finite question bank by constructing confidence intervals with valid frequentist coverage. It introduces Factorized Active Querying (FAQ), which combines a history-informed Bayesian factor model, a hybrid adaptive sampling policy, and Proactive Active Inference (PAI) to efficiently query a new model and produce tight, coverage-guaranteed CIs. The authors prove a martingale CLT-based result for the PAI estimator and demonstrate up to 5× effective sample size gains over baselines across two benchmark suites under varying historical-data missingness, with robust empirical coverage. They also release their datasets and code to enable reproducible benchmarking and future research in scalable, uncertainty-aware LLM evaluation.

Abstract

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.

Efficient Evaluation of LLM Performance with Statistical Guarantees

TL;DR

This work tackles the problem of reliably evaluating large language models under a fixed, finite question bank by constructing confidence intervals with valid frequentist coverage. It introduces Factorized Active Querying (FAQ), which combines a history-informed Bayesian factor model, a hybrid adaptive sampling policy, and Proactive Active Inference (PAI) to efficiently query a new model and produce tight, coverage-guaranteed CIs. The authors prove a martingale CLT-based result for the PAI estimator and demonstrate up to 5× effective sample size gains over baselines across two benchmark suites under varying historical-data missingness, with robust empirical coverage. They also release their datasets and code to enable reproducible benchmarking and future research in scalable, uncertainty-aware LLM evaluation.

Abstract

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Proactive Active Inference -- a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.
Paper Structure (23 sections, 4 theorems, 55 equations, 10 figures)

This paper contains 23 sections, 4 theorems, 55 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Factor Model: Extracting Information from History
  5. Proactive Active Inference
  6. An Oracle-Optimal Sequential Sampling Policy
  7. An Active-Learning-Driven Sampling Policy
  8. A Practical Hybrid Sampling Policy
  9. Experimental Setup
  10. Empirical Results
  11. Discussion and Conclusion
  12. Proofs and other Formulas
  13. Full Historical Factor Model Negative Log-Likelihood Objective
  14. Derivation of Online Logistic Regression Laplace Approximation Update
  15. Proof of Theorem \ref{['thm:martingale-clt']}
  16. ...and 8 more sections

Key Result

Theorem 3.1

Consider a sequence of problems (indexed by $m$) such that $n_b\uparrow \infty$ as $m \rightarrow \infty$ and $n_b \leq N_q$ for every $m$. Then (a) $\hat{\theta}_{n_b}$ is unbiased for $\theta$ for every $m$; and (b) under mild regularity conditions (variance stabilization/control and Lindeberg; As where

Figures (10)

  • Figure 1: High-level overview of Factorized Active Querying (FAQ).(a) Using partially-observed historical outcomes $H$, we fit a historical model and question factors (Eq. \ref{['eq:factor-model-objective']}). For a new model, we initialize its latent factor from the historical fit. Then, at each sampling round $t$, we (b) compute the hybrid sampling policy $q_t(\cdot)$ from the current factor posterior (Eq. \ref{['eq:final-hybrid-sampling-policy']}), (c) sample a question $I_t \sim q_t(\cdot)$, and (d) update the model factor using the observed outcome $z_{I_t}$ (Eqs. \ref{['eq:bayesian-factor-model-update-1']}-\ref{['eq:bayesian-factor-model-update-3']}). Finally, we (e) output the Proactive Active Inference (PAI) estimate and confidence interval (Eqs. \ref{['eq:proposed-estimator']}, \ref{['eq:estimator-variance']}), which guarantees model-free, frequentist coverage for finite-bank accuracy.
  • Figure 2: Evaluating $2.2$K LLMs on MMLU-Pro and BBH+GPQA+IFEval+MATH+MuSR with fully-observed historical data. (Top Row) Effective sample size (ESS) vs. budget for FAQ, the strongest baseline per budget (selected post-hoc, favoring baselines), and uniform sampling. Numbers annotate the ESS multiplier relative to uniform sampling. For example, on MMLU-Pro at budget $1500$, an ESS multiplier of $5.01$ means FAQ matches uniform sampling's CI width at budget $7515$ (a $5.01\times$ gain). (Bottom Row) Empirical coverage of the corresponding $95\%$ normal-approximation CIs, averaged over $2.2$K test models and $100$ seeds. Standard errors are shown, but negligible on the plot scale. Higher ESS and ESS-multiplier values indicate better performance.
  • Figure 3: ESS under missing historical data on BBH+GPQA+IFEval+MATH+MuSR. ESS vs. budget for FAQ, the strongest baseline per budget (selected post-hoc), and uniform sampling, per missingness setting. Missingness is parameterized by $n_{\text{full-obs}}$ fully-observed historical rows (out of $2.2$K) and MCAR entrywise observation probability $p_{\text{obs}}$ on the remaining rows (e.g., $n_{\text{full-obs}} = 50$ and $p_{\text{obs}} = 0.1$$\sim$$12\%$ observed). Standard errors are shown, but negligible on the plot scale. MMLU-Pro and coverage results are in Appendix \ref{['appendix:additional-results']}.
  • Figure 4: Per-model coverages on BBH+GPQA+IFEval+MATH+MuSR with fully-observed historical data at budget $7.5\%$. Per-model coverages (averaged over $100$ seeds) vs. (Left) model release date and (Right) true model accuracy. Gray dots: individual models; solid-blue curve: locally-weighted mean with a $K=501$ nearest-neighbor Gaussian kernel; light-blue band: local $\pm 1$ SD band.
  • Figure 5: CI-width ratio (post-hoc best traditional active inference ablation / FAQ) across budgets/missingness. Missingness uses $n_{\text{full-obs}}=0$ and MCAR $p_{\text{obs}}$ (legend values) on the remaining entries. Standard errors are shown, but negligible at the plot scale. Width ratios $> 1$ favor FAQ (narrower CIs).
  • ...and 5 more figures

Theorems & Definitions (6)

  • Theorem 3.1
  • Theorem 3.2
  • Proposition 1.4: Martingale differences and unbiasedness
  • proof
  • Lemma 1.5
  • proof