Polynomial Chaos Expanded Gaussian Process

TL;DR

Problem: global GP models with stationary covariance struggle to capture local non-stationarity. Approach: Polynomial Chaos Expanded Gaussian Process (PCEGP) computes input-dependent lengthscale $l(x)$ and noise variance $\\sigma_n^2(x)$ via polynomial chaos expansion and combines multiple covariance functions into a non-stationary kernel $k_\\Sigma$, with hyperparameters optimized through Bayesian optimization using the Tree-Structured Parzen Estimator and Nadam. Contributions: interpretable hyperparameters as analytic polynomials, explicit non-stationarity modeling, heteroscedastic noise estimation, and competitive benchmark performance. Impact: enables robust, interpretable non-stationary regression in engineering and scientific modeling without resorting to ensembles of local models or deep architectures.

Abstract

In complex and unknown processes, global models are initially generated over the entire experimental space but often fail to provide accurate predictions in local areas. A common approach is to use local models, which requires partitioning the experimental space and training multiple models, adding significant complexity. Recognizing this limitation, this study addresses the need for models that effectively represent both global and local experimental spaces. It introduces a novel machine learning (ML) approach: Polynomial Chaos Expanded Gaussian Process (PCEGP), leveraging polynomial chaos expansion (PCE) to calculate input-dependent hyperparameters of the Gaussian process (GP). This provides a mathematically interpretable approach that incorporates non-stationary covariance functions and heteroscedastic noise estimation to generate locally adapted models. The model performance is compared to different algorithms in benchmark tests for regression tasks. The results demonstrate low prediction errors of the PCEGP, highlighting model performance that is often competitive with or better than previous methods. A key advantage of the presented model is its interpretable hyperparameters along with training and prediction runtimes comparable to those of a GP.

Figures (12)

• Figure 1: Visualization of the tree-structured Parzen estimator (TPE) process for Bayesian hyperparameter optimization
• Figure 2: Evaluation of the Polynomial Chaos Expanded Gaussian Process with data scaling, point-dependent hyperparameter calculation, hyperparameter transformation, and output prediction
• Figure 3: Runtime comparison for training and prediction of PCEGP and GP
• Figure 4: Comparison of PCEGP and GP prediction quality for modeling the non-stationary Higdon function
• Figure 5: Comparison of PCEGP and GP prediction quality for modeling the non-stationary Higdon function