Investigating Bayesian optimization for expensive-to-evaluate black box functions: Application in fluid dynamics

TL;DR

This work tackles optimizing expensive-to-evaluate black-box objectives, common in CFD and hyperparameter tuning, by systematically comparing Gaussian-process-based Bayesian optimization algorithms across analytical, Monte Carlo, and batched acquisitions. It clarifies the roles of a $GP$ surrogate and a suite of acquisition functions, including $PI$, $EI$, $UCB$ (with $\beta$), and $MES$, and demonstrates how batching and Monte Carlo methods enable parallel evaluations. The study finds that optimistic acquisition strategies (notably $GP$-UBC variants) are robust on complex landscapes, while Monte Carlo and multi-point approaches offer viable, parallelizable alternatives without heavily sacrificing performance; these insights hold even with limited initial data and moderate noise. Two CFD-inspired applications illustrate practical impact: Bayesian optimization identifies control profiles that yield drag reduction and net energy savings with only a handful of expensive evaluations, underscoring the method's potential for real-world, resource-intensive engineering problems.

Abstract

Bayesian optimization provides an effective method to optimize expensive-to-evaluate black box functions. It has been widely applied to problems in many fields, including notably in computer science, e.g. in machine learning to optimize hyperparameters of neural networks, and in engineering, e.g. in fluid dynamics to optimize control strategies that maximize drag reduction. This paper empirically studies and compares the performance and the robustness of common Bayesian optimization algorithms on a range of synthetic test functions to provide general guidance on the design of Bayesian optimization algorithms for specific problems. It investigates the choice of acquisition function, the effect of different numbers of training samples, the exact and Monte Carlo based calculation of acquisition functions, and both single-point and multi-point optimization. The test functions considered cover a wide selection of challenges and therefore serve as an ideal test bed to understand the performance of Bayesian optimization to specific challenges, and in general. To illustrate how these findings can be used to inform a Bayesian optimization setup tailored to a specific problem, two simulations in the area of computational fluid dynamics are optimized, giving evidence that suitable solutions can be found in a small number of evaluations of the objective function for complex, real problems. The results of our investigation can similarly be applied to other areas, such as machine learning and physical experiments, where objective functions are expensive to evaluate and their mathematical expressions are unknown.

Figures (15)

• Figure 1: Different challenges and levels of complexity of the simulations represented by the shapes of four test functions.
• Figure 2: Performance plots for analytical single-point acquisition functions with five initial starting points per input dimension. Solid lines represent the mean over the 50 runs while the shaded area represents the 95% confidence intervals.
• Figure 3: Performance plots for analytical single-point acquisition functions with one initial starting point per input dimension. Solid lines represent the mean over the 50 runs while the shaded area represents the 95% confidence intervals.
• Figure 4: Performance plots for analytical single-point acquisition functions with ten initial starting points per input dimension. Solid lines represent the mean over the 50 runs while the shaded area represents the 95% confidence intervals.
• Figure 5: Performance plots for Monte Carlo single-point acquisition functions with five initial starting points per input dimension. Solid lines represent the mean over the 50 runs while the shaded areas represent the 95% confidence intervals.