Multi-Variable Batch Bayesian Optimization in Materials Research: Synthetic Data Analysis of Noise Sensitivity and Problem Landscape Effects

TL;DR

It is found that the effects of noise depend on the problem landscape: noise degrades the optimization results of a needle-in-a-haystack search (Ackley) dramatically more and with increasing noise, with increasing noise, an increasing probability of landing on the local optimum in Hartmann.

Abstract

Bayesian Optimization (BO) machine learning method is increasingly used to guide experimental optimization tasks in materials science. To emulate the large number of input variables and noise-containing results in experimental materials research, we perform batch BO simulation of six design variables with a range of noise levels. Two test cases relevant for materials science problems are examined: a needle-in-a-haystack case (Ackley function) that may be encountered in, e.g., molecule optimizations, and a smooth landscape with a local optimum in addition to the global optimum (Hartmann function) that may be encountered in, e.g., material composition optimization. We show learning curves, performance metrics, and visualization to effectively track the optimization progression and evaluate how the optimization outcomes are affected by noise, batch-picking method, choice of acquisition function, and exploration hyperparameter values. We find that the effects of noise depend on the problem landscape: noise degrades the optimization results of a needle-in-a-haystack search (Ackley) dramatically more. However, with increasing noise, we observe an increasing probability of landing on the local optimum in Hartmann. Therefore, prior knowledge of the problem domain structure and noise level is essential when designing BO for materials research experiments. Synthetic data studies -- with known ground truth and controlled noise levels -- enable us to isolate and evaluate the impact of different batch BO components, {\it e.g.}, acquisition policy, objective metrics, and hyperparameter values, before transitioning to the inherent uncertainties of real experimental systems. The results and methodology of this study will facilitate a greater utilization of BO in guiding experimental materials research, specifically in settings with a large number of design variables to optimize.

Figures (12)

• Figure 1: : Visualization of the 3D representation of the ground truth of (a) Ackley function, where x1,x2 variables are projected in 3D representation, and of (b) Hartmann function, where x1,x3 variables are projected in 3D representation. All the other input variables are chosen so that the maximum A( X) for a given (x1, x2) is shown for Ackley and the maximum H( X) for a given (x1, x3) is shown for Hartmann. The global maximum is labeled as the ‘GT max’ (cyan cross for Ackley, black cross for Hartmann) both functions and Hartmann function 2nd maxima labeled as the '2nd True Max' (red star).
• Figure 2: The workflow in the batch BO benchmarking: (a) LHS of the 6D input variable space to pick the starting points for the BO, (b) evaluations of the analytical test function at the selected points with an option to include noise, (c) the surrogate GPR model training at each iteration of the BO learning, (d) picking a batch of input points for the next iteration. The whole BO learning runs 50 iterations to generate the (e) X (top) and y (bottom) learning curves of the benchmark criteria selected for this work, tracking the distance of the surrogate model optimum point to the true optimum location and the value of the surrogate model optimum, respectively. This whole process is repeated for 99 different LHS initial samplings to collect statisticsBO.
• Figure 3: BO results using UCB with $\beta$ = 1 on noise-free Ackley function. Learning curve in (a) $||{\bf X}^* - {\bf X}_{\rm max} ||$ and (b) $\mu_{\rm D} ({\bf X}^*)$ for all 99 LHS initial starts. The 25th percentile (green triangle), 50th percentile (red circle) and 75th percentile (blue triangle) regions are highlighted to exemplify poor, median, and good LHS BO models, respectively. (c) Visualization of the 3D representation (x1 vs. x2) for the 50th percentile BO model, showing how BO iterations zero in on ${\bf X}_{\rm max}$ (cyan cross). Light blue circles are the 24 initial LHS selections and the pink to dark red points are training points (progressively darker). (d) Zoomed-in 2D heat map for variables x1 and x2 within the range [-2, 2] near ${\bf X}_{\rm max}$. (e) Parity plot of the 50th percentile BO model prediction vs. GT values for all 224 sampled points (LHS + 50 iteration at a batch size of 4).
• Figure 4: (a)-(f) 3D representations of Ackley test function (UCB with $\beta$=1) at different iterations. Blue points are the initial LHS points. Light pink points are sampled at earlier iterations and the dark red points are sampled at later iterations according to the color bar.
• Figure 5: BO results using UCB with $\beta$ = 1 on noise-free Hartmann function. Learning curve in (a) $||{\bf X}^* - {\bf X}_{\rm max} ||$ and (b) $\mu_{\rm D} ({\bf X}^*)$ for 99 LHS BO models. The 25th percentile (green triangle), 50th percentile (red circle) and 75th percentile (blue triangle) regions are highlighted to exemplify poor, median, and good LHS BO models, respectively. (c) Visualization of the 3D representations for (x1 vs. x2) variables pair, for the 50th percentile BO model at the last iteration, showing how BO iterations zero in on ${\bf X}_{\rm max}$. Red circles are the 24 initial LHS selections and the blue to light green points are training points (blue progressively to green).(d) Parity plot of the 50th percentile BO model prediction vs. GT values for all 224 sampled points (LHS + 50 iteration at a batch size of 4).