Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics

Niki van Stein, Anna V. Kononova, Haoran Yin, Thomas Bäck

TL;DR

This work addresses the lack of standardized benchmarks for evaluating LLM-driven Automated Algorithm Discovery in continuous black-box optimisation. It introduces BLADE, a modular benchmarking suite that integrates diverse problem sets (MA-BBOB, SBOX-COST), configurable experiment design, LLM interfaces, and analysis tools (Code Evolution Graphs, AOCC, EAF, ELO) to enable capability-focused evaluation. The paper defines capabilities such as generalisation, problem-class specialization, and information exploitation, and demonstrates two use-cases—mutation-prompt analysis and cross-LLM comparison—showing that prompt strategies and LLM choice can materially affect the quality of discovered solvers, sometimes matching or exceeding human-designed baselines like CMA-ES. The framework is open-source with full prompts and results, supporting reproducible, robust evaluation and guiding future benchmark design and tool integration in LLM-driven AAD.

Abstract

The application of Large Language Models (LLMs) for Automated Algorithm Discovery (AAD), particularly for optimisation heuristics, is an emerging field of research. This emergence necessitates robust, standardised benchmarking practices to rigorously evaluate the capabilities and limitations of LLM-driven AAD methods and the resulting generated algorithms, especially given the opacity of their design process and known issues with existing benchmarks. To address this need, we introduce BLADE (Benchmark suite for LLM-driven Automated Design and Evolution), a modular and extensible framework specifically designed for benchmarking LLM-driven AAD methods in a continuous black-box optimisation context. BLADE integrates collections of benchmark problems (including MA-BBOB and SBOX-COST among others) with instance generators and textual descriptions aimed at capability-focused testing, such as generalisation, specialisation and information exploitation. It offers flexible experimental setup options, standardised logging for reproducibility and fair comparison, incorporates methods for analysing the AAD process (e.g., Code Evolution Graphs and various visualisation approaches) and facilitates comparison against human-designed baselines through integration with established tools like IOHanalyser and IOHexplainer. BLADE provides an `out-of-the-box' solution to systematically evaluate LLM-driven AAD approaches. The framework is demonstrated through two distinct use cases exploring mutation prompt strategies and function specialisation.

BLADE: Benchmark suite for LLM-driven Automated Design and Evolution of iterative optimisation heuristics

TL;DR

This work addresses the lack of standardized benchmarks for evaluating LLM-driven Automated Algorithm Discovery in continuous black-box optimisation. It introduces BLADE, a modular benchmarking suite that integrates diverse problem sets (MA-BBOB, SBOX-COST), configurable experiment design, LLM interfaces, and analysis tools (Code Evolution Graphs, AOCC, EAF, ELO) to enable capability-focused evaluation. The paper defines capabilities such as generalisation, problem-class specialization, and information exploitation, and demonstrates two use-cases—mutation-prompt analysis and cross-LLM comparison—showing that prompt strategies and LLM choice can materially affect the quality of discovered solvers, sometimes matching or exceeding human-designed baselines like CMA-ES. The framework is open-source with full prompts and results, supporting reproducible, robust evaluation and guiding future benchmark design and tool integration in LLM-driven AAD.

Abstract

The application of Large Language Models (LLMs) for Automated Algorithm Discovery (AAD), particularly for optimisation heuristics, is an emerging field of research. This emergence necessitates robust, standardised benchmarking practices to rigorously evaluate the capabilities and limitations of LLM-driven AAD methods and the resulting generated algorithms, especially given the opacity of their design process and known issues with existing benchmarks. To address this need, we introduce BLADE (Benchmark suite for LLM-driven Automated Design and Evolution), a modular and extensible framework specifically designed for benchmarking LLM-driven AAD methods in a continuous black-box optimisation context. BLADE integrates collections of benchmark problems (including MA-BBOB and SBOX-COST among others) with instance generators and textual descriptions aimed at capability-focused testing, such as generalisation, specialisation and information exploitation. It offers flexible experimental setup options, standardised logging for reproducibility and fair comparison, incorporates methods for analysing the AAD process (e.g., Code Evolution Graphs and various visualisation approaches) and facilitates comparison against human-designed baselines through integration with established tools like IOHanalyser and IOHexplainer. BLADE provides an `out-of-the-box' solution to systematically evaluate LLM-driven AAD approaches. The framework is demonstrated through two distinct use cases exploring mutation prompt strategies and function specialisation.
Paper Structure (13 sections, 5 figures, 1 table)

This paper contains 13 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Capability-Focused Benchmarking
  5. Real world applications
  6. Use-cases
  7. Use-case: Mutation Prompts
  8. Experimental Setup
  9. AAD Evaluation Results
  10. Validation Against Human-Designed Baseline
  11. Use-case: LLM Comparison
  12. Reproducibility and Transparency
  13. Conclusions and Outlook

Figures (5)

  • Figure 1: Distribution of the fitness (AOCC) of algorithms resulting from each AAD run. The AOCC is obtained by evaluating each of the resulting algorithms ($10$ algorithms per LLaMEA configuration) on $50$ evaluation instances of MA-BBOB and $10$ random seeds.
  • Figure 2: Code Evolution Graphs of $3$ runs (columns) per LLaMEA variant (rows) showing how the number of Python tokens (y-axis) changes per generated algorithm (x-axis). The colour of the nodes denotes the AOCC score (yellow is higher and better, blue is lower and worse), while connections between nodes denote parent-offspring relations and the size of nodes encodes the number of connections produced from this node. Runs are sometimes showing fewer than $100$ evaluations due to runtime errors inside the generated algorithms (resulting in a negative fitness which is not displayed).
  • Figure 3: Empirical Attainment Function (EAF) curves of the best solutions found per LLaMEA configuration (best of $10$ runs) and a CMA-ES baseline. Each algorithm is tested on $50$ different MA-BBOB instances and $10$ independent runs.
  • Figure 4: ELO rating (higher is better) of the best solutions found per LLaMEA configuration (best of $10$ runs) and a CMA-ES baseline. Each algorithm is tested on $50$ different MA-BBOB instances and $10$ independent runs. ELO rating is calculated using a tournament of $100\,000$ one against one comparisons.
  • Figure 5: Fitness (AOCC) distribution of all final solutions generated using LLaMEA with different LLMs.