Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

Aashish Anantha Ramakrishnan, Ardavan Saeedi, Hamid Reza Hassanzadeh, Fazlolah Mohaghegh, Dongwon Lee

Abstract

With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.

Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation

Abstract

With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.
Paper Structure (13 sections, 8 equations, 3 figures, 9 tables)

This paper contains 13 sections, 8 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Definition
  5. Generative Active Testing: An Overview
  6. Experiments
  7. Findings and Discussion
  8. Conclusion
  9. Ablation Studies
  10. Differentiation of Active Testing from Active Learning
  11. Dataset Statistics
  12. Baseline Acquisition Functions
  13. Prompt Templates and Evaluation Settings

Figures (3)

  • Figure 1: An overview of the components of the Generative Active Testing Framework.
  • Figure 2: Step-wise flow diagram detailing tasks performed by the Statement Adaptation Module.
  • Figure 3: Variance and Average Estimation Error. Results across acquisition runs on the PubMedQA dataset. Experimental settings: $\mathcal{F}_{\text{main}} = \text{Llama 3.1 8B}$, $\mathcal{F}_{\text{surrogate}} = \text{NLIVerifier}$, and $\mathcal{F}_{\text{oracle}} = \text{RunnerUp}$.