Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Algorithm-Selection and Performance-Prediction via Learning Discriminating Training Samples

Quentin Renau, Emma Hart

TL;DR

This work tackles algorithm-selection and performance prediction in continuous optimization by contrasting feature-based methods with a meta-algorithm that tunes a simple SA to produce short, discriminatory trajectories. Using irace to optimize SA hyper-parameters, the approach yields input trajectories that improve classification and regression performance on the BBOB suite while using far fewer evaluations than ELA features. Across LOIO validation, trajectory-based models often outperform feature-based ones, and transfer-learning experiments suggest that tuned trajectories can generalize across solvers to reduce computation further. The results support a scalable, algorithm-centric data paradigm for selection and prediction, with potential extensions to larger portfolios and diverse benchmarks.

Abstract

The choice of input-data used to train algorithm-selection models is recognised as being a critical part of the model success. Recently, feature-free methods for algorithm-selection that use short trajectories obtained from running a solver as input have shown promise. However, it is unclear to what extent these trajectories reliably discriminate between solvers. We propose a meta approach to generating discriminatory trajectories with respect to a portfolio of solvers. The algorithm-configuration tool irace is used to tune the parameters of a simple Simulated Annealing algorithm (SA) to produce trajectories that maximise the performance metrics of ML models trained on this data. We show that when the trajectories obtained from the tuned SA algorithm are used in ML models for algorithm-selection and performance prediction, we obtain significantly improved performance metrics compared to models trained both on raw trajectory data and on exploratory landscape features.

Improving Algorithm-Selection and Performance-Prediction via Learning Discriminating Training Samples

TL;DR

This work tackles algorithm-selection and performance prediction in continuous optimization by contrasting feature-based methods with a meta-algorithm that tunes a simple SA to produce short, discriminatory trajectories. Using irace to optimize SA hyper-parameters, the approach yields input trajectories that improve classification and regression performance on the BBOB suite while using far fewer evaluations than ELA features. Across LOIO validation, trajectory-based models often outperform feature-based ones, and transfer-learning experiments suggest that tuned trajectories can generalize across solvers to reduce computation further. The results support a scalable, algorithm-centric data paradigm for selection and prediction, with potential extensions to larger portfolios and diverse benchmarks.

Abstract

The choice of input-data used to train algorithm-selection models is recognised as being a critical part of the model success. Recently, feature-free methods for algorithm-selection that use short trajectories obtained from running a solver as input have shown promise. However, it is unclear to what extent these trajectories reliably discriminate between solvers. We propose a meta approach to generating discriminatory trajectories with respect to a portfolio of solvers. The algorithm-configuration tool irace is used to tune the parameters of a simple Simulated Annealing algorithm (SA) to produce trajectories that maximise the performance metrics of ML models trained on this data. We show that when the trajectories obtained from the tuned SA algorithm are used in ML models for algorithm-selection and performance prediction, we obtain significantly improved performance metrics compared to models trained both on raw trajectory data and on exploratory landscape features.
Paper Structure (16 sections, 4 figures, 2 tables)

This paper contains 16 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Reproducibility:
  3. Related Work
  4. Methods
  5. Searching for Discriminatory Trajectories
  6. Data
  7. Machine Learning Tasks
  8. Inputs for ML models
  9. Models
  10. Validation
  11. Results
  12. Tuned SA Parameters
  13. Algorithm Selection
  14. Performance Prediction
  15. Transfer Learning to reduce computation
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Distributions of SA parameters found by irace for the different ML tasks: classification, performance prediction of CMA-ES, PSO, and DE.
  • Figure 2: Accuracy of classification on the LOIO cross-validation for best-so-far and current probing-trajectories, time series features and time series feature selection for $2$ and $7$ generations. Median ELA feature accuracy is represented by lines for $300$ and $500$ function evaluations.
  • Figure 3: RMSE of performance prediction of CMA-ES, PSO, and DE on the LOIO cross-validation for best-so-far and current probing-trajectories for $7$ generations. Median ELA feature RMSE is represented by lines for $300$ and $500$ function evaluations.
  • Figure 4: RMSE of performance prediction of CMA-ES on the LOIO cross-validation for best-so-far and current probing-trajectories for all tuned SA (including other tasks). Median ELA features RMSE are represented by lines for $300$ and $500$ function evaluations.