The Pitfalls of Benchmarking in Algorithm Selection: What We Are Getting Wrong

TL;DR

This paper identifies two major pitfalls in evaluating algorithm selection meta-models: flawed leave-instance-out (LIO) evaluation that can exploit spurious correlations within problem classes, and the use of scale-sensitive performance metrics that misrepresent AS capability. Through controlled experiments on the COCO benchmark with multiple feature sets (including non-informative and class-based features) and a rank-focused evaluation via PRE, the authors show that LIO can yield misleadingly low PRE values due to correlations rather than genuine predictivity, while scale-sensitive targets like target precision do not reliably translate into better algorithm selection when evaluated with scale-free metrics. They demonstrate that a single scale-related feature can dominate scale-sensitive evaluations, but this advantage vanishes when using rank-based metrics, underscoring the need for appropriate baselines and evaluation criteria. The work concludes with practical recommendations to improve AS benchmarking and emphasizes that more rigorous, context-appropriate evaluation is essential for trustworthy claims about meta-model effectiveness.

Abstract

Algorithm selection, aiming to identify the best algorithm for a given problem, plays a pivotal role in continuous black-box optimization. A common approach involves representing optimization functions using a set of features, which are then used to train a machine learning meta-model for selecting suitable algorithms. Various approaches have demonstrated the effectiveness of these algorithm selection meta-models. However, not all evaluation approaches are equally valid for assessing the performance of meta-models. We highlight methodological issues that frequently occur in the community and should be addressed when evaluating algorithm selection approaches. First, we identify flaws with the "leave-instance-out" evaluation technique. We show that non-informative features and meta-models can achieve high accuracy, which should not be the case with a well-designed evaluation framework. Second, we demonstrate that measuring the performance of optimization algorithms with metrics sensitive to the scale of the objective function requires careful consideration of how this impacts the construction of the meta-model, its predictions, and the model's error. Such metrics can falsely present overly optimistic performance assessments of the meta-models. This paper emphasizes the importance of careful evaluation, as loosely defined methodologies can mislead researchers, divert efforts, and introduce noise into the field

Figures (3)

• Figure 1: PRE of different meta-models with the LIO evaluation strategy (top) and LPO evaluation strategy (bottom). Each dot represents one train/test split of the data.
• Figure 2: Comparison of target precision and rank metrics (y-axis) for quantifying the performance of algorithms for the first instance of COCO problems 4, 13, and 24. All instances are artificially rescaled and their scale is measured with $f_{scale}$ feature (x-axis). Results are reported for only two out of five algorithms in the portfolio.
• Figure 3: Left: MSE calculated between true and predicted target precision for two meta-models aggregated across all five optimization algorithms. With only a single feature, the rf-precision meta-model outperforms the mean-precision by orders of magnitude. Right: Evaluating four different meta-models to determine their success in ranking five optimization algorithms in the LPO setting. Two of the meta-models directly predict the ranks, while the remaining two meta-models predict target precision, which is then transformed into ranks. The large differences between the meta-models are now nonexistent.