Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Explainable Benchmarking for Iterative Optimization Heuristics

Niki van Stein, Diederick Vermetten, Anna V. Kononova, Thomas Bäck

TL;DR

The IOHxplainer software library is introduced, for systematic analysing the performance of various optimization algorithms and the impact of their different components and hyperparameters, aiming to improve future benchmarking and algorithm design practices.

Abstract

Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This paper presents a novel approach we call explainable benchmarking. Introducing the IOH-Xplainer software framework, for analyzing and understanding the performance of various optimization algorithms and the impact of their different components and hyper-parameters. We showcase the framework in the context of two modular optimization frameworks. Through this framework, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design.

Explainable Benchmarking for Iterative Optimization Heuristics

TL;DR

The IOHxplainer software library is introduced, for systematic analysing the performance of various optimization algorithms and the impact of their different components and hyperparameters, aiming to improve future benchmarking and algorithm design practices.

Abstract

Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This paper presents a novel approach we call explainable benchmarking. Introducing the IOH-Xplainer software framework, for analyzing and understanding the performance of various optimization algorithms and the impact of their different components and hyper-parameters. We showcase the framework in the context of two modular optimization frameworks. Through this framework, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design.
Paper Structure (25 sections, 1 equation, 9 figures, 10 tables)

This paper contains 25 sections, 1 equation, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Modular Optimization Heuristics
  4. Explainable AI for EC
  5. Explainable Benchmarking
  6. Existing methodologies
  7. Visual test
  8. Benchmarking Function Groups
  9. Domain Specific Benchmarks
  10. Modern benchmarking practices that lack immediate explainability
  11. IOHxplainer
  12. Robustness to Instance Variation and Random Seed Variation
  13. Explainable Benchmarking of 52 128 Algorithm Configurations
  14. Used Algorithms
  15. Data collection setup
  16. ...and 10 more sections

Figures (9)

  • Figure 1: The proposed IOHxplainer framework, for the automatic and explainable analysis of algorithm components and hyperparameters.
  • Figure 2: Hyper-parameter contributions per benchmark function for d=5 for modular CMA-ES. Categorical hyperparameters are encoded to integer values in alphabetic order where NaN is encoded as $-1$. Refer to Table \ref{['tab:modde_modules']} for the colour coding used.
  • Figure 3: Hyper-parameter contributions per benchmark function for d=30 for modular CMA-ES. Categorical hyperparameters are encoded to integer values in alphabetic order where NaN is encoded as $-1$. Refer to Table \ref{['tab:modde_modules']} for the colour coding used.
  • Figure 4: Hyper-parameter contributions to the AOCC (shap values) per benchmark function for d=5 for modular DE. Options per module are sorted alphabetically (for categorical parameters) or numerically, refer to Table \ref{['tab:modde_modules']} for colour coding.
  • Figure 5: Hyper-parameter contributions per benchmark function for d=30 for modular DE. Options per module are sorted alphabetically (for categorical parameters) or numerically, refer to Table \ref{['tab:modde_modules']} for colour coding.
  • ...and 4 more figures