Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LABCAT: Locally adaptive Bayesian optimization using principal-component-aligned trust regions

E. Visser, C. E. van Daalen, J. C. Schoeman

TL;DR

LABCAT advances Bayesian optimization by introducing two locally adaptive mechanisms: length-scale–based rescaling of observations and a principal-component–aligned rotation of the trust region. These transforms, combined with approximate GP hyperparameter updates and a greedy observation discarding strategy within a fixed trust region, yield improved conditioning and faster convergence for non-stationary or ill-conditioned objectives. Empirical results on synthetic benchmarks and the COCO BBOB suite show LABCAT outperforming standard BO and many trust-region–based alternatives, particularly on unimodal/high-conditioning problems. The work suggests LABCAT as a practical, scalable approach for expensive black-box optimization with strong potential for extensions to noise, mixed variable types, and gradient-enhanced modeling.

Abstract

Bayesian optimization (BO) is a popular method for optimizing expensive black-box functions. BO has several well-documented shortcomings, including computational slowdown with longer optimization runs, poor suitability for non-stationary or ill-conditioned objective functions, and poor convergence characteristics. Several algorithms have been proposed that incorporate local strategies, such as trust regions, into BO to mitigate these limitations; however, none address all of them satisfactorily. To address these shortcomings, we propose the LABCAT algorithm, which extends trust-region-based BO by adding a rotation aligning the trust region with the weighted principal components and an adaptive rescaling strategy based on the length-scales of a local Gaussian process surrogate model with automatic relevance determination. Through extensive numerical experiments using a set of synthetic test functions and the well-known COCO benchmarking software, we show that the LABCAT algorithm outperforms several state-of-the-art BO and other black-box optimization algorithms.

LABCAT: Locally adaptive Bayesian optimization using principal-component-aligned trust regions

TL;DR

LABCAT advances Bayesian optimization by introducing two locally adaptive mechanisms: length-scale–based rescaling of observations and a principal-component–aligned rotation of the trust region. These transforms, combined with approximate GP hyperparameter updates and a greedy observation discarding strategy within a fixed trust region, yield improved conditioning and faster convergence for non-stationary or ill-conditioned objectives. Empirical results on synthetic benchmarks and the COCO BBOB suite show LABCAT outperforming standard BO and many trust-region–based alternatives, particularly on unimodal/high-conditioning problems. The work suggests LABCAT as a practical, scalable approach for expensive black-box optimization with strong potential for extensions to noise, mixed variable types, and gradient-enhanced modeling.

Abstract

Bayesian optimization (BO) is a popular method for optimizing expensive black-box functions. BO has several well-documented shortcomings, including computational slowdown with longer optimization runs, poor suitability for non-stationary or ill-conditioned objective functions, and poor convergence characteristics. Several algorithms have been proposed that incorporate local strategies, such as trust regions, into BO to mitigate these limitations; however, none address all of them satisfactorily. To address these shortcomings, we propose the LABCAT algorithm, which extends trust-region-based BO by adding a rotation aligning the trust region with the weighted principal components and an adaptive rescaling strategy based on the length-scales of a local Gaussian process surrogate model with automatic relevance determination. Through extensive numerical experiments using a set of synthetic test functions and the well-known COCO benchmarking software, we show that the LABCAT algorithm outperforms several state-of-the-art BO and other black-box optimization algorithms.
Paper Structure (34 sections, 73 equations, 5 figures, 13 tables, 1 algorithm)

This paper contains 34 sections, 73 equations, 5 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related literature
  3. Summary of contributions
  4. Structure of the paper
  5. Preliminaries
  6. Gaussian process regression
  7. Trust-region-based Bayesian optimization
  8. Overview of the LABCAT algorithm
  9. Length-scale-based rescaling
  10. Principal-component-based rotation
  11. The LABCAT algorithm
  12. Combined observation transformation
  13. GP hyperparameter estimation
  14. Trust region definition and observation discarding
  15. Algorithm initialization and termination
  16. ...and 19 more sections

Figures (5)

  • Figure 1: A flowchart of the LABCAT algorithm. The primary added components of the LABCAT algorithm, compared to the standard trust-region-based BO described in Alg. \ref{['alg:BO_opt']}, is given by the shaded areas. A full mathematical description of the LABCAT algorithm in given in Alg. \ref{['alg:algpa']}.
  • Figure 2: A visualization of enforcing the invariant properties (described by \ref{['li:rescale_invariant_1']}--\ref{['li:output_invariant']}) on a number of observations from an arbitrary function, where the observed output values are represented using a colour map. The data is (\ref{['subfig:length_X_Y']}) centred on the minimum candidate (marked with a $+$), the output values are normalized and the most likely length-scales ($\ell^{*}_{1}$, $\ell^{*}_{2}$) for a GP fitted to the data are shown. Using these length-scales, (\ref{['subfig:transformed_X_Y']}) the input data is rescaled such that these length-scales equal unity, with all invariant properties now preserved.
  • Figure 3: A visualization of enforcing the invariant properties (described by \ref{['li:rescale_invariant_1']}, \ref{['li:output_invariant']} and \ref{['li:rotate_invariant']}) on a number of observations from an arbitrary function, where the observed output values are represented using a colour map. The data is (\ref{['subfig:centered_X_Y']}) centred on the minimum candidate (marked with a $+$), the output values are normalized and the weighted principal components are shown. Using these principal components, (\ref{['subfig:rotated_X_Y']}) the input data is rotated such that these principal components are aligned with the coordinate axes, with all invariant properties now preserved.
  • Figure 4: Illustrative example showing a typical run of a hypothetical trust-region-based BO algorithm applied to the 2-D Rosenbrock function \ref{['subfig:illus_no_pca']} without and \ref{['subfig:illus_pca']} with weighted principal component trust region rotation. A subset of trust regions (indicated in black) centred on the respective minimum candidate solutions (indicated in red) are given, the global optimum is indicated by the magenta cross at $(1.0, 1.0)$ and observations other than the minimum candidate are not indicated to maintain visual clarity.
  • Figure 5: Performance of selected algorithms applied to synthetic 2-D test functions. We conduct 50 independent optimization runs per algorithm with a sampling budget of 150 objective function evaluations. The mean and standard deviation, indicated by the shaded regions, of the logarithmic global regret, which is the log-difference between the best candidate solution $y_{\min}$ at each sampling iteration of the objective function and the global minimum $f_{\min}$, are reported. The definitions and domains of each objective function is given in \ref{['subsec-app:synth_bench_funcs']}.