Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Active Evaluation Acquisition for Efficient LLM Benchmarking

Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, Graham Horwood

TL;DR

This work investigates strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy, and introduces a novel RL-based policy that leverages the captured dependencies.

Abstract

As large language models (LLMs) become increasingly versatile, numerous large scale benchmarks have been developed to thoroughly assess their capabilities. These benchmarks typically consist of diverse datasets and prompts to evaluate different aspects of LLM performance. However, comprehensive evaluations on hundreds or thousands of prompts incur tremendous costs in terms of computation, money, and time. In this work, we investigate strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy. Our approach models the dependencies across test examples, allowing accurate prediction of the evaluation outcomes for the remaining examples based on the outcomes of the selected ones. Consequently, we only need to acquire the actual evaluation outcomes for the selected subset. We rigorously explore various subset selection policies and introduce a novel RL-based policy that leverages the captured dependencies. Empirical results demonstrate that our approach significantly reduces the number of evaluation prompts required while maintaining accurate performance estimates compared to previous methods.

Active Evaluation Acquisition for Efficient LLM Benchmarking

TL;DR

This work investigates strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy, and introduces a novel RL-based policy that leverages the captured dependencies.

Abstract

As large language models (LLMs) become increasingly versatile, numerous large scale benchmarks have been developed to thoroughly assess their capabilities. These benchmarks typically consist of diverse datasets and prompts to evaluate different aspects of LLM performance. However, comprehensive evaluations on hundreds or thousands of prompts incur tremendous costs in terms of computation, money, and time. In this work, we investigate strategies to improve evaluation efficiency by selecting a subset of examples from each benchmark using a learned policy. Our approach models the dependencies across test examples, allowing accurate prediction of the evaluation outcomes for the remaining examples based on the outcomes of the selected ones. Consequently, we only need to acquire the actual evaluation outcomes for the selected subset. We rigorously explore various subset selection policies and introduce a novel RL-based policy that leverages the captured dependencies. Empirical results demonstrate that our approach significantly reduces the number of evaluation prompts required while maintaining accurate performance estimates compared to previous methods.
Paper Structure (22 sections, 9 equations, 4 figures, 7 tables, 2 algorithms)

This paper contains 22 sections, 9 equations, 4 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation
  4. Modeling Dependencies via Neural Processes
  5. Evaluation Acquisition Policy
  6. Random Policy
  7. Static Policy
  8. Dynamic Policy
  9. Cold Start Problem
  10. Related Works
  11. Experiments
  12. Ablation Studies
  13. Conclusion
  14. Neural Process
  15. Architecture
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Experiment results on five popular LLM evaluation benchmarks, with shaded areas indicating the standard deviation over three runs.
  • Figure 2: Evaluate the situation with model bias, where test models are from different model families compared to the training models.
  • Figure 3: Evaluate the cold start problem on MMLU benchmark, where 15 subsets are left out as cold start prompts.
  • Figure A.1: The architecture of the neural process model.