Scaling Up Active Testing to Large Language Models

TL;DR

This work tackles the challenge of label-efficient evaluation for large language models by identifying and mitigating three main computational bottlenecks in active testing: surrogate-model training, surrogate-prediction, and target-model prediction costs. It introduces practical cost-saving measures, notably constructing a fixed, small surrogate via a single in-context learning step and optionally avoiding target-model predictions entirely, to enable scalable data acquisition. The authors show that sampling-based active testing with these surrogates substantially improves risk estimation over uniform random sampling (often reducing error by 25-50%), and they provide a bootstrap-based estimator to gauge risk-estimation accuracy in a single run. Collectively, the approach enables more accurate, economical evaluation of LLMs and offers a diagnostic tool to assess when active testing is performing well in practical deployments.

Abstract

Active testing enables label-efficient evaluation of predictive models through careful data acquisition, but it can pose a significant computational cost. We identify cost-saving measures that enable active testing to be scaled up to large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without making predictions with the target model. As a result we are able to achieve much more accurate evaluations of LLM performance relative to using randomly acquired data. We additionally introduce a bootstrap estimator of evaluation error, which we show to be a useful indicator of how well active testing is working within a single run.

Figures (12)

• Figure 1: Our proposed active-testing approach enables low-error estimates of the risk (expected predictive loss) of a large language model (here the 70B version of Llama 2) on four text-classification problems (SST-2, FPB, HS and Subj) while only using a small label budget. We compare uniform-random testing against active testing (LURE), using either a 7B or 70B surrogate model with in-context learning to guide data acquisition.
• Figure 2: Cheap surrogate models support effective data acquisition for active testing. We compare uniform-random testing to active testing (LURE) across four datasets. To guide active data acquisition we use a surrogate model that we train using a single step of in-context learning and then keep fixed. This stripped-back surrogate-model training, along with the use of small surrogate models relative to the target models, allows us to drastically reduce the computational cost of active testing while achieving strong performance.
• Figure 3: Older Llama 2 models are useful surrogates for active testing of newer Gemma 3 target models.
• Figure 4: Our sampling-based active testing (LURE) approach works more reliably than interpolation-based active testing (ASE) when using cheaply constructed surrogate models. In ASE the surrogate model not only guides data acquisition but also much more directly affects the risk estimate: the labels used to compute the expected loss of the target model are simulated from the surrogate model.
• Figure 5: Using the surrogate model to approximate the target model during data acquisition (here this means acquiring data based on the predictive entropy of the surrogate model) reduces the need for target-model predictions while still performing well.