Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BayeSQP: Bayesian Optimization through Sequential Quadratic Programming

Paul Brunzema, Sebastian Trimpe

TL;DR

BayeSQP addresses high-dimensional black-box optimization with constraints by integrating classical sequential quadratic programming (SQP) into a Bayesian framework. It uses second-order Gaussian process surrogates that jointly model the objective and constraints, including function values, gradients, and Hessians, to form an uncertainty-aware SQP subproblem that is tractable as a second-order cone program. A one-dimensional line search is performed via constrained posterior sampling to select feasible, high-probability-improvement steps. Empirically, BayeSQP outperforms state-of-the-art BO methods in high-dimensional constrained settings and maintains practical runtime advantages, while also highlighting sensitivity to initialization and kernel choices. This framework effectively bridges traditional optimization techniques with modern probabilistic surrogates for robust, scalable black-box optimization.

Abstract

We introduce BayeSQP, a novel algorithm for general black-box optimization that merges the structure of sequential quadratic programming with concepts from Bayesian optimization. BayeSQP employs second-order Gaussian process surrogates for both the objective and constraints to jointly model the function values, gradients, and Hessian from only zero-order information. At each iteration, a local subproblem is constructed using the GP posterior estimates and solved to obtain a search direction. Crucially, the formulation of the subproblem explicitly incorporates uncertainty in both the function and derivative estimates, resulting in a tractable second-order cone program for high probability improvements under model uncertainty. A subsequent one-dimensional line search via constrained Thompson sampling selects the next evaluation point. Empirical results show thatBayeSQP outperforms state-of-the-art methods in specific high-dimensional settings. Our algorithm offers a principled and flexible framework that bridges classical optimization techniques with modern approaches to black-box optimization.

BayeSQP: Bayesian Optimization through Sequential Quadratic Programming

TL;DR

BayeSQP addresses high-dimensional black-box optimization with constraints by integrating classical sequential quadratic programming (SQP) into a Bayesian framework. It uses second-order Gaussian process surrogates that jointly model the objective and constraints, including function values, gradients, and Hessians, to form an uncertainty-aware SQP subproblem that is tractable as a second-order cone program. A one-dimensional line search is performed via constrained posterior sampling to select feasible, high-probability-improvement steps. Empirically, BayeSQP outperforms state-of-the-art BO methods in high-dimensional constrained settings and maintains practical runtime advantages, while also highlighting sensitivity to initialization and kernel choices. This framework effectively bridges traditional optimization techniques with modern probabilistic surrogates for robust, scalable black-box optimization.

Abstract

We introduce BayeSQP, a novel algorithm for general black-box optimization that merges the structure of sequential quadratic programming with concepts from Bayesian optimization. BayeSQP employs second-order Gaussian process surrogates for both the objective and constraints to jointly model the function values, gradients, and Hessian from only zero-order information. At each iteration, a local subproblem is constructed using the GP posterior estimates and solved to obtain a search direction. Crucially, the formulation of the subproblem explicitly incorporates uncertainty in both the function and derivative estimates, resulting in a tractable second-order cone program for high probability improvements under model uncertainty. A subsequent one-dimensional line search via constrained Thompson sampling selects the next evaluation point. Empirical results show thatBayeSQP outperforms state-of-the-art methods in specific high-dimensional settings. Our algorithm offers a principled and flexible framework that bridges classical optimization techniques with modern approaches to black-box optimization.
Paper Structure (56 sections, 1 theorem, 23 equations, 10 figures, 4 tables)

This paper contains 56 sections, 1 theorem, 23 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Preliminaries
  4. Sequential quadratic programming
  5. Gaussian processes
  6. Related work
  7. Scalable Bayesian optimization
  8. Local Bayesian optimization
  9. Bayesian optimization and Gaussian processes in classical optimization
  10. BayeSQP: Merging classic SQP and Bayesian optimization
  11. Second-order Gaussian processes as surrogate models for BayeSQP
  12. Deriving the subproblem for BayeSQP
  13. Expected value SQP subproblem
  14. Uncertainty-aware SQP subproblem
  15. Tractability through joint Gaussian process
  16. ...and 41 more sections

Key Result

Corollary 1

The solution for the search direction of BSUB is equivalent to solution of eq:eqp for $\delta_f = 0.5$ and $\delta_c = 0.5$. (Proof in Appendix sec:proof_cor1)

Figures (10)

  • Figure 1: Overview.BayeSQP combines ideas from sequential quadratic programming and Bayesian optimization for efficient high-dimensional black-box optimization.
  • Figure 2: The power of Gaussian processes. Although we only have zeroth-order information about the function, the differentiability of the GP allows us to estimate both the gradient and curvature. All estimates provided are in expectation; the associated uncertainties are not shown.
  • Figure 3: Intuition on optimization behavior of BayeSQP. Disregarding uncertainty ($\delta_f, \delta_c=0.5$, left) results in directions tangential to the circular constraint, while a for a very conservative configuration ($\delta_f,\delta_c=0.05$, center right), the constraint acts as a repellent to ensure feasibility. Values in between (center left) yield a desirable convergence path to the optimum. On the right, we see the space-filling behavior of constrained logEI which is fundamentally different compared to the local BayeSQP.
  • Figure 4: Unconstrained within-model comparison. As the dimensions grow, the benefit of local search increases,with BayeSQP significantly outperforming the other baselines. Note that for SAASBO, no runs completed within the 24-hour time cap when the dimensionality exceeded 32.
  • Figure 5: Constrained within-model comparison.BayeSQP demonstrates superior performance at high dimensions, fast optimization times, as well as low sensitivity to parameter choice.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Corollary 1: Recovering the expected value formulation
  • Remark 1