Beyond Regrets: Geometric Metrics for Bayesian Optimization

TL;DR

Four new geometric metrics are proposed that allow us to compare Bayesian optimization algorithms considering the geometry of both query points and global optima, or query points, and validate that these metrics can provide more delicate interpretation of Bayesian optimization, on top of assessment via the conventional metrics.

Abstract

Bayesian optimization is a principled optimization strategy for a black-box objective function. It shows its effectiveness in a wide variety of real-world applications such as scientific discovery and experimental design. In general, the performance of Bayesian optimization is reported through regret-based metrics such as instantaneous, simple, and cumulative regrets. These metrics only rely on function evaluations, so that they do not consider geometric relationships between query points and global solutions, or query points themselves. Notably, they cannot discriminate if multiple global solutions are successfully found. Moreover, they do not evaluate Bayesian optimization's abilities to exploit and explore a search space given. To tackle these issues, we propose four new geometric metrics, i.e., precision, recall, average degree, and average distance. These metrics allow us to compare Bayesian optimization algorithms considering the geometry of both query points and global optima, or query points. However, they are accompanied by an extra parameter, which needs to be carefully determined. We therefore devise the parameter-free forms of the respective metrics by integrating out the additional parameter. Finally, we validate that our proposed metrics can provide more delicate interpretation of Bayesian optimization, on top of assessment via the conventional metrics.

Figures (30)

• Figure 1: Examples of how precision and recall are computed where three global optima (red x) and ten query points (green +) are given. Selected points (blue dot) are marked if they are located in the vicinity (yellow circle) of query points. The Branin function is used for these examples and each colorbar depicts its function values.
• Figure 2: Results versus iterations for the Branin function. Sample means over 50 rounds and the standard errors of the sample mean over 50 rounds are depicted. PF stands for parameter-free.
• Figure 3: Results versus iterations for the Zakharov 16D function. Sample means over 50 rounds and the standard errors of the sample mean over 50 rounds are depicted. PF stands for parameter-free.
• Figure 4: Spearman's rank correlation coefficients between metrics for different benchmark functions such as the Branin and Zakharov 16D functions. Red regions indicate the coefficients with NaN values. More results are depicted in Figures \ref{['fig:correlations_models_appendix_1']}, \ref{['fig:correlations_models_appendix_2']}, and \ref{['fig:correlations_models_appendix_3']}.
• Figure 5: Spearman's rank correlation coefficients between metrics for different Bayesian optimization algorithms with the expected improvement.