SEEK: Self-adaptive Explainable Kernel For Nonstationary Gaussian Processes

TL;DR

SEEK addresses nonstationarity in Gaussian Processes by learning a self-adaptive, neuron-inspired kernel that combines a set of base kernels with input-dependent weights and a bias, then passes their sum through a nonlinear activation to guarantee kernel validity. By leveraging kernel-closure properties and differentiable learnable components, SEEK achieves flexible, interpretable nonstationarity while maintaining positive semi-definiteness. Across seven benchmarks and engineering datasets, SEEK consistently improves mean predictions and uncertainty quantification, especially in low-to-mid data regimes, and demonstrates robustness to design choices and optimization stability. The work suggests promising extensions to sparse approximations, multi-layer architectures, and learnable activations, with potential impact on Bayesian optimization and reliable uncertainty estimation in complex systems.

Abstract

Gaussian processes (GPs) are powerful probabilistic models that define flexible priors over functions, offering strong interpretability and uncertainty quantification. However, GP models often rely on simple, stationary kernels which can lead to suboptimal predictions and miscalibrated uncertainty estimates, especially in nonstationary real-world applications. In this paper, we introduce SEEK, a novel class of learnable kernels to model complex, nonstationary functions via GPs. Inspired by artificial neurons, SEEK is derived from first principles to ensure symmetry and positive semi-definiteness, key properties of valid kernels. The proposed method achieves flexible and adaptive nonstationarity by learning a mapping from a set of base kernels. Compared to existing techniques, our approach is more interpretable and much less prone to overfitting. We conduct comprehensive sensitivity analyses and comparative studies to demonstrate that our approach is not only robust to many of its design choices, but also outperforms existing stationary/nonstationary kernels in both mean prediction accuracy and uncertainty quantification.

Figures (6)

• Figure 1: Schematic illustration of our kernel: SEEK has a set of weighted base kernels with learnable hyperparameters where the weights are learnable functions. These weighted kernels, along with a learnable bias term, are summed and then passed through an appropriate activation function to produce the final nonstationary covariance function.
• Figure 2: Prediction of a GP using SEEK on the Analytic I problem, along with two of the learned weighted base kernels and the resulting covariance function evaluated at two reference points.
• Figure 3: Sensitivity studies: We conduct four sensitivity studies on two benchmark problems to evaluate the impact of key design choices on the performance of SEEK. The results are based on 16 repetitions.
• Figure 4: Our proposed kernel SEEK with (a) 1 and (b) 6 Gaussian kernels as base kernels on the Volcano dataset: Increasing the number of base kernels reduces error and improves confidence, as reflected in the colorbars (same ranges are used in (a) and (b) to facilitate direct comparison). Black dots represent the training data points.
• Figure 5: Comparative studies: We compare SEEK against other nonstationary kernels on four benchmark problems with varying degrees of nonlinearity and dimensionality.

Theorems & Definitions (2)

• Theorem 1: Kernel Validity under Analytic Transformations horn2012matrix
• Definition 1: SEEK Kernel