Landscape-Aware Automated Algorithm Configuration using Multi-output Mixed Regression and Classification

TL;DR

Overall, configurations with better performance can be best identified by using NN models trained on a combination of RGF and MA-BBOB functions, which can most of the time outperform the off-the-shelf default configuration considered by practitioners with limited knowledge about AAC.

Abstract

In landscape-aware algorithm selection problem, the effectiveness of feature-based predictive models strongly depends on the representativeness of training data for practical applications. In this work, we investigate the potential of randomly generated functions (RGF) for the model training, which cover a much more diverse set of optimization problem classes compared to the widely-used black-box optimization benchmarking (BBOB) suite. Correspondingly, we focus on automated algorithm configuration (AAC), that is, selecting the best suited algorithm and fine-tuning its hyperparameters based on the landscape features of problem instances. Precisely, we analyze the performance of dense neural network (NN) models in handling the multi-output mixed regression and classification tasks using different training data sets, such as RGF and many-affine BBOB (MA-BBOB) functions. Based on our results on the BBOB functions in 5d and 20d, near optimal configurations can be identified using the proposed approach, which can most of the time outperform the off-the-shelf default configuration considered by practitioners with limited knowledge about AAC. Furthermore, the predicted configurations are competitive against the single best solver in many cases. Overall, configurations with better performance can be best identified by using NN models trained on a combination of RGF and MA-BBOB functions.

Figures (6)

• Figure 1: An overview of our proposed landscape-aware AAC approach that can identify optimal configurations for BBO problems, consisting of a training and testing phase. During the training phase, using a preferably large set of RGF, the respective ELA features and optimal configurations identified through HPO (performed on RGF) are utilized to train NN models. The pre-trained models can then be deployed to predict the best suited configuration for unseen BBO problems based on their ELA features in the testing phase.
• Figure 2: The optimization convergence of $500$ configurations evaluated using HPO on three chosen RGF. The x-axis shows the number of function evaluations, while the y-axis shows the re-scaled objective values, with $0$ being the best solution found in all runs. Each curve represents a configuration run using modular CMA-ES (median over $10$ repetitions). (Left) Ideal for AAC purposes, where a clear ranking of configurations is possible. (Middle) Ambiguous ranking of algorithm configurations, where all configurations are equally competitive. (Right) The global optimum seems to be an outlier that can only be found by a few configurations.
• Figure 3: An example of the architecture of a dense NN model. From left to right, an input layer, three hidden layers, and several output layers, with one output layer for regression and four layers for classification tasks.
• Figure 4: Projection of the ELA feature space to a $2d$ visualization using t-distributed stochastic neighbor embedding (t-SNE) tsne_maaten2008 for $1\,000$ RGF, $1\,000$ MA-BBOB, and $24$ BBOB functions in $5d$ (left) and $20d$ (right), using a similar approach as in long2024generating.
• Figure 5: Performance of modular CMA-ES using different configurations for $24$ BBOB functions in $5d$, each repeated for $10$ times. The AUC is computed based on objective values min-max normalized using the global optimum and worst solution in all configurations, divided by the evaluation budget. A lower AUC is better.