Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Partial Rankings of Optimizers

Julian Rodemann, Hannah Blocher

TL;DR

This work introduces a framework for benchmarking optimizers according to multiple criteria over various test functions based on a recently introduced union-free generic depth function for partial orders/rankings that fully exploits the ordinal information and allows for incomparability.

Abstract

We introduce a framework for benchmarking optimizers according to multiple criteria over various test functions. Based on a recently introduced union-free generic depth function for partial orders/rankings, it fully exploits the ordinal information and allows for incomparability. Our method describes the distribution of all partial orders/rankings, avoiding the notorious shortcomings of aggregation. This permits to identify test functions that produce central or outlying rankings of optimizers and to assess the quality of benchmarking suites.

Partial Rankings of Optimizers

TL;DR

This work introduces a framework for benchmarking optimizers according to multiple criteria over various test functions based on a recently introduced union-free generic depth function for partial orders/rankings that fully exploits the ordinal information and allows for incomparability.

Abstract

We introduce a framework for benchmarking optimizers according to multiple criteria over various test functions. Based on a recently introduced union-free generic depth function for partial orders/rankings, it fully exploits the ordinal information and allows for incomparability. Our method describes the distribution of all partial orders/rankings, avoiding the notorious shortcomings of aggregation. This permits to identify test functions that produce central or outlying rankings of optimizers and to assess the quality of benchmarking suites.
Paper Structure (11 sections, 4 equations, 4 figures)

This paper contains 11 sections, 4 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Results
  4. URM Statement
  5. Acknowledgements
  6. Preliminary Definitions
  7. Definition of the union-free generic depth function on posets and some computational aspects
  8. Results on BBOB Suite
  9. Results on Multi-Objective Evolutionary Algorithms
  10. Outlook: Design and Curation of Benchmarking Suites
  11. Related Work

Figures (4)

  • Figure 1: Orderings of optimizers corresponding to highest ($0.65$, left) and lowest ($0.29$, right) ufg depth.
  • Figure 2: BBOB suite based on dimension 2: Poset corresponding to the maximal (0.21, left) and minimal (0.11, right) ufg depth value.
  • Figure 3: BBOB suite based on all dimensions: Three posets with the highest depth value have this dominance order in common, i.e., RANDOMSEARCH being dominated by all other optimizers is true for the three most central posets.
  • Figure 4: Multi-objective evolutionary algorithms: Orderings of optimizers corresponding to highest ($0.39$, left) and lowest ($0.17$, right) ufg depth.

Theorems & Definitions (3)

  • Example B.1
  • Definition B.1
  • Example B.2