BONO-Bench: A Comprehensive Test Suite for Bi-objective Numerical Optimization with Traceable Pareto Sets

TL;DR

BONO-Bench tackles the challenge of reliable bi-objective benchmarking by introducing a scalable generator that produces traceable Pareto sets from convex-quadratic building blocks, enabling explicit control over modality, Pareto shapes, conditioning, and discretization. It combines bottom-up and composite construction ideas to yield 20 problem classes with known optima and a Pareto-front-approximation mechanism, implemented in the bonobench Python package for reproducible benchmarking. The study demonstrates the generator’s capabilities and provides an evaluation pipeline using standard MOEAs and a random baseline across varying dimensionalities, revealing insights on front shapes, discretization effects, and algorithm performance. The work advances the state of benchmarking by offering exact target values for Pareto-compliant indicators (R2 and HV) and a principled, extensible framework for generating diverse, controllable bi-objective test problems with traceable fronts, with broad implications for algorithm selection, configuration, and comparative research in multi-objective optimization.

Abstract

The evaluation of heuristic optimizers on test problems, better known as \emph{benchmarking}, is a cornerstone of research in multi-objective optimization. However, most test problems used in benchmarking numerical multi-objective black-box optimizers come from one of two flawed approaches: On the one hand, problems are constructed manually, which result in problems with well-understood optimal solutions, but unrealistic properties and biases. On the other hand, more realistic and complex single-objective problems are composited into multi-objective problems, but with a lack of control and understanding of problem properties. This paper proposes an extensive problem generation approach for bi-objective numerical optimization problems consisting of the combination of theoretically well-understood convex-quadratic functions into unimodal and multimodal landscapes with and without global structure. It supports configuration of test problem properties, such as the number of decision variables, local optima, Pareto front shape, plateaus in the objective space, or degree of conditioning, while maintaining theoretical tractability: The optimal front can be approximated to an arbitrary degree of precision regarding Pareto-compliant performance indicators such as the hypervolume or the exact R2 indicator. To demonstrate the generator's capabilities, a test suite of 20 problem categories, called \emph{BONO-Bench}, is created and subsequently used as a basis of an illustrative benchmark study. Finally, the general approach underlying our proposed generator, together with the associated test suite, is publicly released in the Python package \texttt{bonobench} to facilitate reproducible benchmarking.

Figures (17)

• Figure 1: Different landscape visualizations of the same bi-objective test problem (instance 10 of the two-dimensional FewSpheres problem, cf. \ref{['sec:bono_multi_without']}).
• Figure 2: Illustration of the perturbation of a quadratic function with other quadratic functions. The more perturbation functions are used, the smaller the difference becomes.
• Figure 3: Illustration of our Pareto front approximation technique with the hypervolume indicator. The parametrized fronts of the peak combinations are illustrated as dashed lines, and actually known/evaluated points are represented by crosses. Subfigure (a) shows the process of subdividing the front between two evaluated points $y_l$ and $y_r$. The original approximation error $\varepsilon_I(y_l,y_r)$ is given by the lighter shaded area, while the remaining error after the addition of $y_m$ is given by the darker shaded area for $\varepsilon_I(y_l,y_m)$, and $\varepsilon_I(y_m,y_r)$, respectively. Subfigure (b) shows the interactions when multiple peaks contribute to the front: Approximation errors are computed for each front separately, though pairs of consecutive points whose ideal point is dominated by evaluated points are excluded from the further process. The next pair of points to subdivide is decided greedily by the largest gap, and illustrated here by the dotted area.
• Figure 4: Illustration of instances of the unimodal, axis-aligned problems from the BONO-Bench suite, which show log-scaled landscape visualizations of the individual objectives (left) as well as PLOT schaepermeier2020plot visualizations of the decision and objective space (right). The upper (sphere) problem features no conditioning, while the lower (ellipsoid) problem is highly conditioned ($\kappa \approx 2.4 \cdot10^5$).
• Figure 5: Ellipsoids with linear Pareto set, but different Pareto front shapes. The particular distance parameters sampled are $p \approx 2.89$, $p = 1$, and $p \approx 0.64$ for the convex, linear and concave fronts, respectively.