Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-fidelity Bayesian Optimization: A Review

Bach Do, Ruda Zhang

TL;DR

MF BO tackles expensive design optimization by marrying multi-fidelity surrogates with Bayesian optimization, leveraging LF information within a GP prior over the HF objective to guide high-cost evaluations. It surveys GP-based MF surrogates (e.g., LMC, KOH autoregressive, GMGP, deep MF GP) and MF acquisition functions (EI, PI, KG, ES, MES, GP-UCB, and their MF extensions) to outline a structured MF BO framework. The review discusses three MF modeling classes (adjustment, composition, input augmentation) and three optimization strategies (derivative-based, model-then-optimize, model-and-optimize), plus extensions to constraints, high dimensionality, uncertainty, and multi-objective problems, with illustrative airfoil optimization and test-function experiments. It also identifies open challenges (computational cost, fidelity selection, scalable inference, and robust design under uncertainty) and points to practical software tools (e.g., Emukit, BoTorch, SMAC3) for implementing MF BO in engineering contexts.

Abstract

Resided at the intersection of multi-fidelity optimization (MFO) and Bayesian optimization (BO), MF BO has found a niche in solving expensive engineering design optimization problems, thanks to its advantages in incorporating physical and mathematical understandings of the problems, saving resources, addressing exploitation-exploration trade-off, considering uncertainty, and processing parallel computing. The increasing number of works dedicated to MF BO suggests the need for a comprehensive review of this advanced optimization technique. In this paper, we survey recent developments of two essential ingredients of MF BO: Gaussian process (GP) based MF surrogates and acquisition functions. We first categorize the existing MF modeling methods and MFO strategies to locate MF BO in a large family of surrogate-based optimization and MFO algorithms. We then exploit the common properties shared between the methods from each ingredient of MF BO to describe important GP-based MF surrogate models and review various acquisition functions. By doing so, we expect to provide a structured understanding of MF BO. Finally, we attempt to reveal important aspects that require further research for applications of MF BO in solving intricate yet important design optimization problems, including constrained optimization, high-dimensional optimization, optimization under uncertainty, and multi-objective optimization.

Multi-fidelity Bayesian Optimization: A Review

TL;DR

MF BO tackles expensive design optimization by marrying multi-fidelity surrogates with Bayesian optimization, leveraging LF information within a GP prior over the HF objective to guide high-cost evaluations. It surveys GP-based MF surrogates (e.g., LMC, KOH autoregressive, GMGP, deep MF GP) and MF acquisition functions (EI, PI, KG, ES, MES, GP-UCB, and their MF extensions) to outline a structured MF BO framework. The review discusses three MF modeling classes (adjustment, composition, input augmentation) and three optimization strategies (derivative-based, model-then-optimize, model-and-optimize), plus extensions to constraints, high dimensionality, uncertainty, and multi-objective problems, with illustrative airfoil optimization and test-function experiments. It also identifies open challenges (computational cost, fidelity selection, scalable inference, and robust design under uncertainty) and points to practical software tools (e.g., Emukit, BoTorch, SMAC3) for implementing MF BO in engineering contexts.

Abstract

Resided at the intersection of multi-fidelity optimization (MFO) and Bayesian optimization (BO), MF BO has found a niche in solving expensive engineering design optimization problems, thanks to its advantages in incorporating physical and mathematical understandings of the problems, saving resources, addressing exploitation-exploration trade-off, considering uncertainty, and processing parallel computing. The increasing number of works dedicated to MF BO suggests the need for a comprehensive review of this advanced optimization technique. In this paper, we survey recent developments of two essential ingredients of MF BO: Gaussian process (GP) based MF surrogates and acquisition functions. We first categorize the existing MF modeling methods and MFO strategies to locate MF BO in a large family of surrogate-based optimization and MFO algorithms. We then exploit the common properties shared between the methods from each ingredient of MF BO to describe important GP-based MF surrogate models and review various acquisition functions. By doing so, we expect to provide a structured understanding of MF BO. Finally, we attempt to reveal important aspects that require further research for applications of MF BO in solving intricate yet important design optimization problems, including constrained optimization, high-dimensional optimization, optimization under uncertainty, and multi-objective optimization.
Paper Structure (61 sections, 100 equations, 14 figures, 7 tables, 6 algorithms)

This paper contains 61 sections, 100 equations, 14 figures, 7 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Multi-fidelity optimization
  3. Surrogate-based optimization
  4. Multi-fidelity surrogates
  5. Adjustment.
  6. Composition.
  7. Input augmentation.
  8. Multi-fidelity optimization strategies
  9. Derivative-based approach.
  10. Model-then-optimize approach.
  11. Model-and-optimize approach.
  12. Multi-fidelity Bayesian optimization
  13. Efficient global optimization
  14. Bayesian optimization
  15. Multi-fidelity Bayesian optimization
  16. ...and 46 more sections

Figures (14)

  • Figure 1: Mathematical modeling for engineering design optimization: (1) fundamental objects, (2) relations, (3) mathematical abstractions, and (4) other concepts in engineering design optimization.
  • Figure 2: (a) Distinction between SbO, MFO, BO, and MF BO; (b) Hierarchy of these optimization concepts.
  • Figure 3: Additive, multiplicative, and hybrid adjustment MF surrogates.
  • Figure 4: Deterministic versus probabilistic adjustment MF surrogates.
  • Figure 5: Illustration of two consecutive iterations (a) and (b) of BO for minimizing a univariate objective function.
  • ...and 9 more figures