Comparative study of regression vs pairwise models for surrogate-based heuristic optimisation

TL;DR

The paper investigates surrogate-based heuristic optimisation under expensive fitness evaluations by comparing surface regression surrogates with a novel pairwise surrogate formulation that uses binary classification. It systematically evaluates five ML models across eight strategy combinations (baseline, probability, quality-distance, and diversity, including their hybrids) within an online DE framework, using SOCO'11 benchmarks and a GECCO'21 industrial problem. Key findings show that performance hinges not only on predictive accuracy but also on bias direction and how predictions govern actual evaluations; the DT/C pairwise surrogate emerges as the most consistently effective configuration, with diversity-driven strategies providing strong gains across most models. The work demonstrates meaningful computational savings, analyzes surrogate contribution over the search process, discusses limitations (e.g., quadratic growth in pairwise data), and suggests future directions for phase-aware and heterogeneous surrogate ensembles.

Abstract

Heuristic optimisation algorithms explore the search space by sampling solutions, evaluating their fitness, and biasing the search in the direction of promising solutions. However, in many cases, this fitness function involves executing expensive computational calculations, drastically reducing the reasonable number of evaluations. In this context, surrogate models have emerged as an excellent alternative to alleviate these computational problems. This paper addresses the formulation of surrogate problems as both regression models that approximate fitness (surface surrogate models) and a novel way to connect classification models (pairwise surrogate models). The pairwise approach can be directly exploited by some algorithms, such as Differential Evolution, in which the fitness value is not actually needed to drive the search, and it is sufficient to know whether a solution is better than another one or not. Based on these modelling approaches, we have conducted a multidimensional analysis of surrogate models under different configurations: different machine learning algorithms (regularised regression, neural networks, decision trees, boosting methods, and random forests), different surrogate strategies (encouraging diversity or relaxing prediction thresholds), and compare them for both surface and pairwise surrogate models. The experimental part of the article includes the benchmark problems already proposed for the SOCO2011 competition in continuous optimisation and a simulation problem included in the recent GECCO2021 Industrial Challenge. This paper shows that the performance of the overall search, when using online machine learning-based surrogate models, depends not only on the accuracy of the predictive model but also on both the kind of bias towards positive or negative cases and how the optimisation uses those predictions to decide whether to execute the actual fitness function.

Figures (13)

• Figure 1: Heat maps of the rankings for each combination of model and strategy for the surface approach (left) and pairwise approach (right).
• Figure 2: Example of the optimisation result per generation (left) or quality function evaluations, f.e. (right). Note: DE is an extended version of the baseline algorithm (with no surrogates) that has been run to reach the same result than the best performing configuration with a surrogate (DT/C).
• Figure 3: Sensitivity, specificity and accuracy for the Hybrid F12 function with the best improvement. Vertical dashed lines define the point when warm up ends.
• Figure 4: Example of the computational cost reduction calculation.
• Figure 5: Distribution of the relative computational cost ratio $\delta e$ for every algorithm and strategy using surface surrogate models.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6