Evolution of Benchmark: Black-Box Optimization Benchmark Design through Large Language Model

TL;DR

This work addresses the bottleneck of hand-crafted Black-Box Optimization benchmarks by introducing Evolution of Benchmark (EoB), an automated, LLM-driven design framework. It casts benchmark design as a bi-objective search over objective programs to maximize landscape similarity ($LSI$) and algorithm-differentiation capability ($ADC$), using a reflection-based co-evolution that couples landscape structure with benchmark code. EoB demonstrates strong performance across classic BBO benchmarking, training of learnable optimizers, and proxy scaffolding for expensive real-world problems, achieving efficient, diverse, and objective benchmarks with minimal human bias. The approach has broad practical impact, enabling scalable, automated benchmark generation, improved evaluation of algorithms, and accelerated development of learning-assisted BBO methods; it also offers a flexible path to extend benchmarks to constrained, multi-objective, and dynamic settings.

Abstract

Benchmark Design in Black-Box Optimization (BBO) is a fundamental yet open-ended topic. Early BBO benchmarks are predominantly human-crafted, introducing expert bias and constraining diversity. Automating this design process can relieve the human-in-the-loop burden while enhancing diversity and objectivity. We propose Evolution of Benchmark (EoB), an automated BBO benchmark designer empowered by the large language model (LLM) and its program evolution capability. Specifically, we formulate benchmark design as a bi-objective optimization problem towards maximizing (i) landscape diversity and (ii) algorithm-differentiation ability across a portfolio of BBO solvers. Under this paradigm, EoB iteratively prompts LLM to evolve a population of benchmark programs and employs a reflection-based scheme to co-evolve the landscape and its corresponding program. Comprehensive experiments validate our EoB is a competitive candidate in multi-dimensional usages: 1) Benchmarking BBO algorithms; 2) Training and testing learning-assisted BBO algorithms; 3) Extending proxy for expensive real-world problems.

Figures (10)

• Figure 1: Paradigm shift from human-crafted benchmark design toward LLM-based automation.
• Figure 2: Visualization of EoB on the co-evolution of the benchmark program and corresponding landscape characteristics.
• Figure 3: Benchmark diversity of CoCo-BBOB and EoB-BBOB.
• Figure 4: Zero-shot generalization performances of four learnable optimizers when trained by different benchmarks. Testing performance curves are presented.
• Figure 5: The ablation results shown as the pareto front set $PF$ found by different baselines.