Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs

TL;DR

The paper tackles the challenge of reliable and cost-efficient evaluation of large language models by introducing a certifiable online evaluation framework. It defines test sample complexity and develops Cer-Eval, a partition-based adaptive algorithm that sequentially selects informative test points to achieve a user-specified estimation error $ε$ with confidence $1-δ$, while providing finite-sample guarantees. Theoretical results establish upper and lower bounds on the required test data and show how variance-aware partitioning can dramatically reduce sample needs under benign partitions. Empirical results on simulations and real benchmarks demonstrate substantial data-efficiency gains (typically 30–40% savings) with robust confidence guarantees, and reveal a connection between model accuracy and evaluation-efficiency. The framework offers a practical, scalable approach to certifiable LLM evaluation and paves the way for tasks such as certified ranking, OOD assessment, and multi-task comparisons.

Abstract

As foundation models continue to scale, the size of trained models grows exponentially, presenting significant challenges for their evaluation. Current evaluation practices involve curating increasingly large datasets to assess the performance of large language models (LLMs). However, there is a lack of systematic analysis and guidance on determining the sufficiency of test data or selecting informative samples for evaluation. This paper introduces a certifiable and cost-efficient evaluation framework for LLMs. Our framework adapts to different evaluation objectives and outputs confidence intervals that contain true values with high probability. We use ``test sample complexity'' to quantify the number of test points needed for a certifiable evaluation and derive tight bounds on test sample complexity. Based on the developed theory, we develop a partition-based algorithm, named Cer-Eval, that adaptively selects test points to minimize the cost of LLM evaluation. Real-world experiments demonstrate that Cer-Eval can save 20% to 40% test points across various benchmarks, while maintaining an estimation error level comparable to the current evaluation process and providing a 95% confidence guarantee.

Figures (9)

• Figure 1: Overview of Cer-Eval, a partition-based adaptive evaluation algorithm. It iterates through four steps until the termination condition is met: (1) partition the dataset based on the evaluated points to minimize evaluation uncertainty, (2) compute summary statistics for each partition, (3) identify the partition that reduces uncertainty the most, and (4) sample and evaluate a new test point from the selected partition.
• Figure 2: Percentage of test points saved by Cer-Eval compared to the baselines on Synthetic Data under (Left) the single partition scenario, (Middle) the easy-to-distinguish scenario, and (Right) the hard-to-distinguish scenario.
• Figure 3: Percentage of test points saved by Cer-Eval compared to baselines in Real-World Benchmarks under GPT-4o: (Left) the MMLU dataset, (Middle) the AlpacaEval dataset, and (Right) the MATH dataset.
• Figure 4: Number of needed test points v.s. model size when evaluating models from multiple families using Cer-Eval, with estimation error level $\epsilon=0.07$ and failure probability $\delta=0.05$.
• Figure 5: Number of needed test points v.s. model accuracy when evaluating models from multiple families using Cer-Eval, with estimation error level $\epsilon=0.07$ and failure probability $\delta=0.05$.

Theorems & Definitions (23)

• Example 3.1
• Definition 3.2: Static evaluation process
• Definition 3.3: Online evaluation process
• Definition 3.4: $(n,\epsilon,\delta)$-certified evaluation algorithm
• Remark 3.5: Practical meaning of certified algorithms
• Definition 3.6: Test sample complexity
• Remark 3.7
• Theorem 4.2
• Definition 5.1: Benign Partition
• Theorem 5.2