An Empirical Study on Ensemble-Based Transfer Learning Bayesian Optimisation with Mixed Variable Types
Natasha Trinkle, Huong Ha, Jeffrey Chan
TL;DR
The paper addresses improving Bayesian optimisation for expensive black-box functions by leveraging historic data through ensemble transfer-learning surrogates. It systematically evaluates initialization, ensemble weighting strategies (including Ridge/Lasso with positive constraints, RGPE, TSTR, and WAC), and mechanisms to handle bad transfer learning across nine benchmarks, including new real-time benchmarks OpenML-CC18 RandomForest, Lassobench, and a cartpole-inspired simulation. Key findings show that warm-start initialisation generally improves convergence and that constraining ensemble weights to be positive yields robust gains, though the best approach is benchmark-dependent; automated standard-vs-transfer-learning switching offers mixed benefits. The work provides practical guidance for TL-BO deployment and introduces a historic-dataset analysis framework using Gower distance and clustering to probe the relationship between minima locations and transfer-learning gains.
Abstract
Bayesian optimisation is a sample efficient method for finding a global optimum of expensive black-box objective functions. Historic datasets from related problems can be exploited to help improve performance of Bayesian optimisation by adapting transfer learning methods to various components of the Bayesian optimisation pipeline. In this study we perform an empirical analysis of various ensemble-based transfer learning Bayesian optimisation methods and pipeline components. We expand on previous work in the literature by contributing some specific pipeline components, and three new real-time transfer learning Bayesian optimisation benchmarks. In particular we propose to use a weighting strategy for ensemble surrogate model predictions based on regularised regression with weights constrained to be positive, and a related component for handling the case when transfer learning is not improving Bayesian optimisation performance. We find that in general, two components that help improve transfer learning Bayesian optimisation performance are warm start initialisation and constraining weights used with ensemble surrogate model to be positive.