Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

Quentin Renau, Emma Hart

TL;DR

This study investigates how the choice of time-series classifier influences algorithm selection when using probing-trajectories from black-box optimisation. By benchmarking 17 classifiers on the BBOB suite with three solvers under LOIO and LOPO cross-validation, it shows that feature-based (Summary) and interval-based (Time Series Forest) models reliably outperform other approaches, including many deep-learning and kernel-based methods. Automated tuning via irace improves performance and these gains transfer across trajectory types, indicating practical robustness. The results suggest that classifier choice is a crucial factor in trajectory-based algorithm selection and provide actionable defaults for practitioners, while highlighting the need to generalise benchmarks to more suites and to consider regression formulations.

Abstract

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.

Algorithm Selection with Probing Trajectories: Benchmarking the Choice of Classifier Model

TL;DR

This study investigates how the choice of time-series classifier influences algorithm selection when using probing-trajectories from black-box optimisation. By benchmarking 17 classifiers on the BBOB suite with three solvers under LOIO and LOPO cross-validation, it shows that feature-based (Summary) and interval-based (Time Series Forest) models reliably outperform other approaches, including many deep-learning and kernel-based methods. Automated tuning via irace improves performance and these gains transfer across trajectory types, indicating practical robustness. The results suggest that classifier choice is a crucial factor in trajectory-based algorithm selection and provide actionable defaults for practitioners, while highlighting the need to generalise benchmarks to more suites and to consider regression formulations.

Abstract

Recent approaches to training algorithm selectors in the black-box optimisation domain have advocated for the use of training data that is algorithm-centric in order to encapsulate information about how an algorithm performs on an instance, rather than relying on information derived from features of the instance itself. Probing-trajectories that consist of a sequence of objective performance per function evaluation obtained from a short run of an algorithm have recently shown particular promise in training accurate selectors. However, training models on this type of data requires an appropriately chosen classifier given the sequential nature of the data. There are currently no clear guidelines for choosing the most appropriate classifier for algorithm selection using time-series data from the plethora of models available. To address this, we conduct a large benchmark study using 17 different classifiers and three types of trajectory on a classification task using the BBOB benchmark suite using both leave-one-instance out and leave-one-problem out cross-validation. In contrast to previous studies using tabular data, we find that the choice of classifier has a significant impact, showing that feature-based and interval-based models are the best choices.
Paper Structure (18 sections, 5 figures, 1 table)

This paper contains 18 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methods
  4. Trajectories
  5. Algorithm Selection
  6. Classification Models
  7. Automated Configuration of Models
  8. Validation Procedure
  9. Results
  10. Default Models on LOIO Cross-Validation
  11. Tuning of Models on LOIO Cross-Validation
  12. Default Models on LOPO Cross-Validation
  13. Insights into model performance
  14. Insights into the difficulty of predicting functions.
  15. Discussion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Accuracy of classification on the LOIO cross-validation for best-so-far and current probing-trajectories for $2$ generations (PSO) and $7$ generations (CMA-ES).
  • Figure 2: Accuracy of classification on the LOIO cross-validation for best-so-far and current probing-trajectories for $2$ generations (DE) and $7$ generations (ALL) for default and tuned models.
  • Figure 3: Heatmap of classification accuracy on the LOPO cross-validation for CMA-ES best-so-far probing-trajectories for $2$ generations for default. $x-$axis represents the functions left out for validation.
  • Figure 4: Number of functions with accuracy above $90\%$ or below $10\%$ for each model.
  • Figure 5: Number of models with accuracy above $90\%$ or below $10\%$ for each function.