Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models

TL;DR

The paper tackles optimization under uncertainty for tuning a scale/precision-type hyperparameter in stochastic models. It introduces a Bayesian optimization framework that builds a probabilistic surrogate for the statistic conditioned on the scale parameter using a Bayesian GLM, which allows analytic evaluation of the objective $f_{\text{true}}(\beta)=\mathbb{E}[|s(\boldsymbol{\omega})-s_0|^2|\beta]$ and yields a closed-form acquisition optimum $\beta^*$. By exploiting a power-law scaling assumption, the method achieves significant data and compute savings (up to about 40x) evidenced in static and dynamic SROM-based engineering problems. This approach enables efficient, real-time hyperparameter tuning in noisy, high-dimensional stochastic settings and offers a path toward scaling to more complex, multi-parameter problems in engineering and science.

Abstract

Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimization under uncertainty can be computationally expensive. In this paper, we present a novel Bayesian optimization framework tailored for hyperparameter tuning under uncertainty, with a focus on optimizing a scale- or precision-type parameter in stochastic models. The proposed method employs a statistical surrogate for the underlying random variable, enabling analytical evaluation of the expectation operator. Moreover, we derive a closed-form expression for the optimizer of the random acquisition function, which significantly reduces computational cost per iteration. Compared with a conventional one-dimensional Monte Carlo-based optimization scheme, the proposed approach requires 40 times fewer data points, resulting in up to a 40-fold reduction in computational cost. We demonstrate the effectiveness of the proposed method through two numerical examples in computational engineering.

Figures (12)

• Figure 1: Conditional model of $s|\beta$. The left panel shows the mean of s as a power function of $\beta$ for positive and negative exponents, with dashed curves indicating the 2.5th and 97.5th conditional percentiles. The right panel presents the corresponding log–log transformations ($\ln s$ vs. $\ln \beta$), where the relationship is linear.
• Figure 2: Static problem: raw data and surrogate model fit. (Left) Raw data is highly noisy with a downward linear trend. Plotted points are 1% of the data, uniformly subsampled. (Right) Bayesian GLM constructed during Bayesian optimization captures the GLM regression function accurately with quantified error, using much fewer data points.
• Figure 3: Static problem: optimization results under different approaches. (Left) Posterior distribution of $(\beta^*, f^*)$ well captures the true statistics. (Right) Histogram of $\beta^*$ posterior samples.
• Figure 4: Static problem: progression of Bayesian optimization. (Left) Sample points during BO iterations. (Right) Evolution of $\beta^*$ posterior distribution and GLM estimate.
• Figure 5: One-dimensional fixed-fixed static system.