Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization

Bach Do, Nafeezat A. Ajenifuja, Taiwo A. Adebiyi, Ruda Zhang

TL;DR

This work tackles the high cost of global sensitivity analysis and optimization with expensive simulators by leveraging Gaussian processes as probabilistic surrogates. It develops and analyzes two posterior-sampling techniques, random Fourier features and pathwise conditioning, offering a scalable route to draw GP samples for function values and for use in GSA and Bayesian optimization (including multi-objective settings). The paper demonstrates how GP-sampled realizations enable Sobol’ index estimation, GP-based Thompson sampling for single- and multi-objective BO, and shows through numerical experiments that these approaches achieve favorable accuracy and computational trade-offs. The results indicate strong potential for applying GP-based sampling to large-scale engineering problems, with practical guidance on when to use each method and how to configure kernels, noise levels, and feature counts; code is made available for replication and extension.

Abstract

High-fidelity simulations and physical experiments are essential for engineering analysis and design, yet their high cost often makes two critical tasks--global sensitivity analysis (GSA) and optimization--prohibitively expensive. This limitation motivates the common use of Gaussian processes (GPs) as proxy regression models that provide uncertainty-aware predictions from a limited number of high-quality observations. GPs naturally enable efficient sampling strategies that support informed decision-making under uncertainty by extracting information from a subset of possible functions for the model of interest. However, direct sampling from GPs is inefficient due to their infinite-dimensional nature and the high cost associated with large covariance matrix operations. Despite their popularity in machine learning and statistics communities, sampling from GPs has received little attention in the community of engineering optimization. In this paper, we present the formulation and detailed implementation of two notable sampling methods--random Fourier features and pathwise conditioning--for generating posterior samples from GPs at reduced computational cost. Alternative approaches are briefly described. Importantly, we detail how the generated samples can be applied in GSA, single-objective optimization, and multi-objective optimization. We show successful applications of these sampling methods through a series of numerical examples.

Sampling from Gaussian Processes: A Tutorial and Applications in Global Sensitivity Analysis and Optimization

TL;DR

This work tackles the high cost of global sensitivity analysis and optimization with expensive simulators by leveraging Gaussian processes as probabilistic surrogates. It develops and analyzes two posterior-sampling techniques, random Fourier features and pathwise conditioning, offering a scalable route to draw GP samples for function values and for use in GSA and Bayesian optimization (including multi-objective settings). The paper demonstrates how GP-sampled realizations enable Sobol’ index estimation, GP-based Thompson sampling for single- and multi-objective BO, and shows through numerical experiments that these approaches achieve favorable accuracy and computational trade-offs. The results indicate strong potential for applying GP-based sampling to large-scale engineering problems, with practical guidance on when to use each method and how to configure kernels, noise levels, and feature counts; code is made available for replication and extension.

Abstract

High-fidelity simulations and physical experiments are essential for engineering analysis and design, yet their high cost often makes two critical tasks--global sensitivity analysis (GSA) and optimization--prohibitively expensive. This limitation motivates the common use of Gaussian processes (GPs) as proxy regression models that provide uncertainty-aware predictions from a limited number of high-quality observations. GPs naturally enable efficient sampling strategies that support informed decision-making under uncertainty by extracting information from a subset of possible functions for the model of interest. However, direct sampling from GPs is inefficient due to their infinite-dimensional nature and the high cost associated with large covariance matrix operations. Despite their popularity in machine learning and statistics communities, sampling from GPs has received little attention in the community of engineering optimization. In this paper, we present the formulation and detailed implementation of two notable sampling methods--random Fourier features and pathwise conditioning--for generating posterior samples from GPs at reduced computational cost. Alternative approaches are briefly described. Importantly, we detail how the generated samples can be applied in GSA, single-objective optimization, and multi-objective optimization. We show successful applications of these sampling methods through a series of numerical examples.

Paper Structure

This paper contains 53 sections, 1 theorem, 67 equations, 21 figures, 3 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Sampling from multivariate and conditional Gaussian distributions
  3. Sampling from multivariate Gaussian distributions
  4. Conditional Gaussian distributions
  5. Two perspectives on Gaussian processes
  6. Function-space view
  7. Weight-space view
  8. Sampling from Gaussian processes
  9. Exhaustive sampling
  10. Random Fourier feature
  11. Pathwise conditioning
  12. Other sampling methods
  13. Orthonormal expansion
  14. Hilbert space approximation
  15. Quasi-Monte Carlo
  16. ...and 38 more sections

Key Result

Theorem 1

(Bochner’s theorem) A continuous, stationary covariance function $\kappa({\bf x},{\bf x}') = \kappa(\boldsymbol{\delta})$ defined over a compact set $\mathcal{X} \subset \mathbb{R}^d$ is positive semi-definite if and only if there exists the Fourier duality of $\kappa(\boldsymbol{\delta})$ and $S(\ where $i$ represents the imaginary unit.

Figures (21)

  • Figure 1: Samples from a bivariate Gaussian distribution $\mathcal{N}\left( {\bf m}, \boldsymbol{\Sigma}\right)$ generated by a composition of affine transformations of samples from the standard Gaussian distribution $\mathcal{N}\left( {\bf 0}, {\bf I}\right)$. (a) Samples from $\mathcal{N}\left( {\bf 0}, {\bf I}\right)$. (b) Samples from $\mathcal{N}\left( {\bf 0}, \boldsymbol{\Sigma}\right)$ obtained by a linear transformation of samples in (a), where the transformation matrix is from the Cholesky decomposition of $\boldsymbol{\Sigma}$. (c) Samples from $\mathcal{N}\left( {\bf m}, \boldsymbol{\Sigma}\right)$ obtained by shifting the samples from (b) in the direction of ${\bf m}$.
  • Figure 2: Approximations of SE and Matérn 5/2 covariance functions using different numbers of random features. (a) SE. (b) Matérn 5/2. (c) Error means (solid lines) and 90% error intervals (shaded areas).
  • Figure 3: Mean and predictive intervals of the exact posterior compared with those of approximations by RFF method with (a) $N_\phi=100$ and (b) $N_\phi=2000$ random features.
  • Figure 4: Illustration of the PC method. (a) A posterior sample is formed by adding an update part to a prior sample. (b) Mean and predictive intervals of the exact posterior compared with those of approximations by the PC method with RFF prior of $N_\phi=2000$.
  • Figure 5: Approximations of the univariate SE covariance function with length scale $l=\sqrt{5}$ and output scale $\sigma_\mathrm{f} = 1$ using (a) Mercer's expansion with $\sigma=\sqrt{3}/2$, (b) Hilbert space approximation, (c) QMC with a Halton sequence, and (d) RFF. (e) Comparison of approximation convergence.
  • ...and 16 more figures

Theorems & Definitions (1)

  • Theorem 1