Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying Easy Instances to Improve Efficiency of ML Pipelines for Algorithm-Selection

Quentin Renau, Emma Hart

TL;DR

The paper tackles the high computational cost of algorithm-selection pipelines by introducing an easy-instance detector that uses cheap probing-trajectories to decide whether to apply a full algorithm-selector or directly solve with a generalist solver. It presents a three-part method consisting of a hardness classifier, an algorithm-selector, and budget-saving strategies that reallocate savings to hard instances in both batch and streaming settings. Using the BBOB/COCO benchmark, the approach achieves budget savings that translate into measurable performance gains relative to a single best solver and, in many cases, outperform the baseline VBS or approach it closely. The results demonstrate practical impact: dynamically reallocating saved budget improves efficiency and solution quality, particularly when curtailing easy runs, while highlighting the importance of robust hardness prediction and budget-management strategies for real-world applications.

Abstract

Algorithm-selection (AS) methods are essential in order to obtain the best performance from a portfolio of solvers over large sets of instances. However, many AS methods rely on an analysis phase, e.g. where features are computed by sampling solutions and used as input in a machine-learning model. For AS to be efficient, it is therefore important that this analysis phase is not computationally expensive. We propose a method for identifying easy instances which can be solved quickly using a generalist solver without any need for algorithm-selection. This saves computational budget associated with feature-computation which can then be used elsewhere in an AS pipeline, e.g., enabling additional function evaluations on hard problems. Experiments on the BBOB dataset in two settings (batch and streaming) show that identifying easy instances results in substantial savings in function evaluations. Re-allocating the saved budget to hard problems provides gains in performance compared to both the virtual best solver (VBS) computed with the original budget, the single best solver (SBS) and a trained algorithm-selector.

Identifying Easy Instances to Improve Efficiency of ML Pipelines for Algorithm-Selection

TL;DR

The paper tackles the high computational cost of algorithm-selection pipelines by introducing an easy-instance detector that uses cheap probing-trajectories to decide whether to apply a full algorithm-selector or directly solve with a generalist solver. It presents a three-part method consisting of a hardness classifier, an algorithm-selector, and budget-saving strategies that reallocate savings to hard instances in both batch and streaming settings. Using the BBOB/COCO benchmark, the approach achieves budget savings that translate into measurable performance gains relative to a single best solver and, in many cases, outperform the baseline VBS or approach it closely. The results demonstrate practical impact: dynamically reallocating saved budget improves efficiency and solution quality, particularly when curtailing easy runs, while highlighting the importance of robust hardness prediction and budget-management strategies for real-world applications.

Abstract

Algorithm-selection (AS) methods are essential in order to obtain the best performance from a portfolio of solvers over large sets of instances. However, many AS methods rely on an analysis phase, e.g. where features are computed by sampling solutions and used as input in a machine-learning model. For AS to be efficient, it is therefore important that this analysis phase is not computationally expensive. We propose a method for identifying easy instances which can be solved quickly using a generalist solver without any need for algorithm-selection. This saves computational budget associated with feature-computation which can then be used elsewhere in an AS pipeline, e.g., enabling additional function evaluations on hard problems. Experiments on the BBOB dataset in two settings (batch and streaming) show that identifying easy instances results in substantial savings in function evaluations. Re-allocating the saved budget to hard problems provides gains in performance compared to both the virtual best solver (VBS) computed with the original budget, the single best solver (SBS) and a trained algorithm-selector.

Paper Structure

This paper contains 19 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Reproducibility:
  3. Related Work
  4. Motivation
  5. Methods
  6. Hardness Classifier
  7. Algorithm-Selection
  8. Saving and Re-allocating budget
  9. Results
  10. Baselines
  11. Re-allocating budget: Batch setting
  12. Algorithm Selection with VBS:
  13. Algorithm Selection with trained selector:
  14. Re-allocating budget: Streaming setting
  15. Algorithm Selection with VBS:
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Algorithm-selection pipeline including the 'easy' instance filter using short trajectories.
  • Figure 2: Median confusion matrix of the hardness prediction.
  • Figure 3: Cumulative difference between the full pipeline using the hardness classification and VBS selector when budget is saved and re-allocated.
  • Figure 4: Boxplots of gains at the end of the stream of instances for each saving budget strategy with VBS selector.
  • Figure 5: Cumulative difference between the full pipeline using the hardness classification and trained selector when budget is saved and re-allocated.
  • ...and 1 more figures