Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms

TL;DR

The paper addresses the limitations of target-based ECDF for benchmarking single-objective black-box optimizers and introduces the empirical attainment function (EAF) as a target-free, information-rich alternative. It formalizes the relationship between ECDF and EAF, showing that the target-based ECDF is the average of the EAF over predefined targets and that the EAF-based ECDF $F_{\widehat{\alpha}}(t)$ emerges as targets become dense; the EAF-based AUC further equates to the mean area over convergence curves (AOCC). The authors implement EAF computation in IOHanalyzer, demonstrate its behavior on the BBOB suite, and propose additional EAF-based statistics such as the Vorob’ev expectation and second-order EAF, with practical implications for ranking and interpretation. Overall, the work provides a principled benchmarking framework that avoids arbitrary target choices and yields richer insights into anytime performance, with concrete tooling and extensions to broader benchmarking contexts.

Abstract

A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We argue that the EAF has several advantages over the target-based ECDF. In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis. We also show that the average area over the convergence curves is a simpler-to-calculate, but equivalent, measure of anytime performance. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform. Finally, we illustrate the use of the EAF via synthetic examples and via the data available for the BBOB suite.

Figures (10)

• Figure 1: Visualization of the EAF corresponding to three runs (red circles, orange diamonds, and blue squares) of an hypothetical single-objective optimizer. For each run, we record as a point, the runtime $t$ (x-axis) at which a new best-so-far objective function value $z$ (y-axis) was found (assuming minimization). The value of the EAF $\widehat{\alpha}$ at point $(t, z)$ is given by the shade of gray specified in the legend. We mark 5 quality targets in the y-axis $\{z_1, z_2, \dots, z_5\}$ for computing the target-based ECDF in Figure \ref{['fig:EAF_ECDF']}.
• Figure 2: Target-based ECDF (solid red line) and EAF-based ECDF (blue dashed line) corresponding to the runs shown in Figure \ref{['fig:EAF_simple']}. The target-based ECDF uses the quality targets $\{z_1, z_2, \dots, z_5\}$ defined in Figure \ref{['fig:EAF_simple']}. For the EAF-based ECDF, we only need $z_\text{min} = z_1$ and $z_\text{max} = z_5$.
• Figure 3: EAF of BFGS over the 24 10-dimensional BBOB functions. The color of a point gives the fraction of runs that reach a given value of $f(x)$ not later than a given number of function evaluations. The black lines delimit the regions with a value (from bottom to top) $\leq 0.25$, $\leq 0.5$ and $\leq 0.75$.
• Figure 4: EAF of CMA-ES over the 24 10-dimensional BBOB functions. See the caption of Figure \ref{['fig:eaf_bfgs']} for more details.
• Figure 5: Differences in the EAFs of CMA-ES and BFGS on BBOB function 14, in 10 dimensions. Blue colors indicate regions where the algorithm in the plot title outperforms the other algorithm.