A Standardized Benchmark Set of Clustering Problem Instances for Comparing Black-Box Optimizers

TL;DR

The paper addresses the need for representative, permutation-invariant benchmarks in continuous black-box optimization by introducing a standardized clustering-based benchmark suite integrated with IOHprofiler. It formalizes the clustering landscape, emphasizes symmetry and neutrality as core challenges, and builds a set of 10 fixed-dimension problem instances derived from PCA-reduced datasets, with baselines provided by repeated K-Means++ runs. A comprehensive analysis—using a large CMA-ES configuration study, comparison to BBOB, and exploratory landscape analysis—demonstrates how these problems differ from traditional suites and how landscape characteristics evolve with dimensionality. The work provides open-source tooling (IOHclustering) and explores symmetry-breaking transformations, offering a path toward more robust and comparable algorithm benchmarking in domains with permutation invariance.

Abstract

One key challenge in optimization is the selection of a suitable set of benchmark problems. A common goal is to find functions which are representative of a class of real-world optimization problems in order to ensure findings on the benchmarks will translate to relevant problem domains. While some problem characteristics are well-covered by popular benchmarking suites, others are often overlooked. One example of such a problem characteristic is permutation invariance, where the search space consists of a set of symmetrical search regions. This type of problem occurs e.g. when a set of solutions has to be found, but the ordering within this set does not matter. The data clustering problem, often seen in machine learning contexts, is a clear example of such an optimization landscape, and has thus been proposed as a base from which optimization benchmarks can be created. In addition to the symmetry aspect, these clustering problems also contain potential regions of neutrality, which can provide an additional challenge to optimization algorithms. In this paper, we present a standardized benchmark suite for the evaluation of continuous black-box optimization algorithms, based on data clustering problems. To gain insight into the diversity of the benchmark set, both internally and in comparison to existing suites, we perform a benchmarking study of a set of modular CMA-ES configurations, as well as an analysis using exploratory landscape analysis. Our benchmark set is open-source and integrated with the IOHprofiler benchmarking framework to encourage its use in future research.

Figures (12)

• Figure 1: Contour plots of selected clustering problems based on 1-dimensional data (visualized as red crosses below the respective plots) with two cluster centers. The two right-most plots are based on the same data as the two left-most plots, but plotted using a wider domain.
• Figure 2: Data underlying each of the 10 problem sets.
• Figure 3: Some example convergence curves (geometric mean) for different population sizes and elitism options (bound correction off, covariance on, $\mu=10$). The dotted line represents the best value found by 100 repetitions of K-Means++.
• Figure 4: EAF over all 10-dimensional problems in our suite, for different population sizes and elitism options (bound correction off, covariance on, $\mu=10$). The EAF bounds are set to the K-Means++ baseline and the worst seen value by all CMA-ES variants, respectively (with a log-scaling between them).
• Figure 5: Relative differences (in terms of function average final function value) between best and worst CMA-ES configurations in the portfolio.