Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models
José Pombal, Nuno M. Guerreiro, Ricardo Rei, André F. T. Martins
TL;DR
Zero-shot Benchmarking (ZSB) introduces a scalable, end-to-end framework that uses language models to automatically generate test data and to judge model outputs, enabling task- and language-agnostic benchmark creation with minimal human input. By employing a meta-prompt for data generation and a judgment prompt for evaluation, ZSB can produce diverse, open-ended benchmarks across text and vision-language modalities, with safety metadata and instance-level documentation. Empirical results across LLM general capabilities, translation, and VLM general capabilities show ZSB rankings align closely with human judgments and often outperform standard static benchmarks, with ablations highlighting the importance of dataset variety and judge-model size. The work demonstrates that benchmarks can co-evolve with model capabilities, remain cost-effective, and be extended to new languages and modalities, while releasing all benchmarks and code to facilitate adoption.
Abstract
As language models improve and become capable of performing more complex tasks across modalities, evaluating them automatically becomes increasingly challenging. Developing strong and robust task-specific automatic metrics gets harder, and human-annotated test sets -- which are expensive to create -- saturate more quickly. A compelling alternative is to design reliable strategies to automate the creation of test data and evaluation, but previous attempts either rely on pre-existing data, or focus solely on individual tasks. We present Zero-shot Benchmarking (ZSB), a framework for creating high-quality benchmarks for any task by leveraging language models for both synthetic test data creation and evaluation. ZSB is simple and flexible: it requires only the creation of a prompt for data generation and one for evaluation; it is scalable to tasks and languages where collecting real-world data is costly or impractical; it is model-agnostic, allowing the creation of increasingly challenging benchmarks as models improve. To assess the effectiveness of our framework, we create benchmarks for five text-only tasks and a multi-modal one: general capabilities in four languages (English, Chinese, French, and Korean), translation, and general vision-language capabilities in English. We then rank a broad range of open and closed systems on our benchmarks. ZSB rankings consistently correlate strongly with human rankings, outperforming widely-adopted standard benchmarks. Through ablations, we find that strong benchmarks can be created with open models, and that judge model size and dataset variety are crucial drivers of performance. We release all our benchmarks, and code to reproduce our experiments and to produce new benchmarks.