LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient

TL;DR

The paper tackles the urgent need for reliable, generic, and efficient LLM benchmarks by introducing an automated evaluation framework with four dimensions and ten criteria to validate benchmark generators. It analyzes direct prompting as a generic benchmark generator and identifies weaknesses in faithfulness, diversity, and controllability, then presents BenchMaker, a modular system that enhances faithfulness (via Stepwise Self-correction and Conflict Guided Discrimination), extends the difficulty boundary (via Diffusion and Strategy Guidance), and boosts diversity (via AttrPrompt and In-batch methods). Empirical results across multiple tasks and 12 LLMs show BenchMaker achieves human-aligned benchmarking quality with strong task alignment and high robustness (Pearson 0.967 with MMLU-Pro) at a fraction of the time and cost ($0.005 per sample; 0.38 minutes), demonstrating broad generalization. The work also provides reliability analyses under noisy labels and ethical considerations, highlighting practical utility and potential societal impact of automated, scalable benchmark generation for NLP research and deployment.

Abstract

The rapid advancement of large language models (LLMs) has led to a surge in both model supply and application demands. To facilitate effective matching between them, reliable, generic and efficient benchmark generators are widely needed. However, human annotators are constrained by inefficiency, and current LLM benchmark generators not only lack generalizability but also struggle with limited reliability, as they lack a comprehensive evaluation framework for validation and optimization. To fill this gap, we first propose an automated and unbiased evaluation framework, structured around four dimensions and ten criteria. Under this framework, we carefully analyze the advantages and weaknesses of directly prompting LLMs as generic benchmark generators. To enhance the reliability, we introduce a series of methods to address the identified weaknesses and integrate them as BenchMaker. Experiments across multiple LLMs and tasks confirm that BenchMaker achieves superior or comparable performance to human-annotated benchmarks on all metrics, highlighting its generalizability and reliability. More importantly, it delivers highly consistent evaluation results across 12 LLMs (0.967 Pearson correlation against MMLU-Pro), while taking only $0.005 and 0.38 minutes per sample.

Figures (7)

• Figure 1: The trends of LLMs released and open-source LLMs downloads per season since the debut of ChatGPT. We obtain the data via the Huggingface API. See details in Appendix \ref{['sec:huggingface']}.
• Figure 2: Pearson correlations among key factors of benchmark evaluation and LLM (Qwen-Plus) judge scores (faithfulness and alignment). The most relevant path of each subject is highlighted in red to show the possible causal chain.
• Figure 3: Overview of BenchMaker.
• Figure 4: Trends of real and labeled difficulty over the index.
• Figure 5: Word cloud of MMLU-Pro and the benchmark generated by BenchMaker under similar assessment demands.