Scale estimation and rate-unbiasedness for Gaussian processes under smoothness misspecification

TL;DR

This work analyzes Gaussian process regression under misspecification of smoothness, introducing rate-unbiasedness as a relaxed criterion for uncertainty quantification. It shows that, when observation locations are quasi-uniform and the model oversmooths, estimating the scale parameter via maximum likelihood or cross-validation yields MSE with the correct order of magnitude without optimising the smoothness. The key theoretical contribution is deriving the asymptotic growth for scale estimators, $\mathbb{E}[\hat{\sigma}_n^2] \asymp n^{2(\nu-\nu_0)/d}$, which combined with $\lVert \mathrm{MSE}_n \rVert_{L^p(D)} \asymp n^{-2\nu/d}$ and $\lVert \mathrm{MSE}_n^* \rVert_{L^p(D)} \asymp n^{-2\nu_0/d}$ yields rate-unbiasedness for $\nu\ge\nu_0$. The paper also extends results to periodic Sobolev kernels and provides extensive numerical experiments showing how smoothness can be inferred from scale-growth and when cross-validation is preferable to ML under misspecification. These findings offer practical guidance for scalable GP uncertainty quantification in misspecified settings, without expensive optimisation over the smoothness parameter.

Abstract

Gaussian process regression is used throughout statistics and machine learning for prediction and uncertainty quantification. A Gaussian process is specified by its mean and covariance functions. Many covariance functions, including Matérns, have a smoothness parameter that is notoriously difficult to specify correctly or estimate from the data. In practice, the smoothness parameter is often selected more or less arbitrarily. We introduce rate-unbiasedness, a relaxed notion of asymptotic optimality which requires that the expected ratio of the mean-square error presumed by a potentially misspecified model and the true, but unknown, mean-square error remain bounded away from zero and infinity as more data are obtained. A rate-unbiased model provides uncertainty quantification that is of correct order of magnitude. We then prove that scale estimation suffices for rate-unbiasedness in a variety of common settings. As estimation of the scale of a Gaussian process is routine and requires no optimisation, rate-unbiasedness can be achieved in many applications.

Figures (7)

• Figure 1: Samples from univariate Gaussian process priors and posteriors defined by the Matérn kernel in \ref{['eq:matern-intro']} with $\nu \in \{1/2, 3/2\}$ and $\lambda = 1$. The red dots are the data points $(x_i, y_i)$, the black line is the posterior mean function, and the shaded region represents the $95\%$ credible intervals around the mean.
• Figure 2: Samples from the Brownian motion ($m=0$), the integrated Brownian motion ($m=1$), and the twice integrated Brownian motion ($m=2$). Covariance kernels of these Gaussian processes are given in \ref{['eq:ibm-kernel']}.
• Figure 3: Samples from the fractional Brownian motion with index $H \in \{0.3, 0.6, 0.9\}$. Covariance kernels of these Gaussian processes are given in \ref{['eq:FBM-kernel']}.
• Figure 4: Samples from Gaussian processes with the periodic covariance kernel in \ref{['eq:korobov-kernel']} with $\alpha \in \{1, 2, 3\}$.
• Figure 5: Smoothness estimator $\hat{\nu}_n$ from \ref{['sec:smoothness-estimation']} (left and middle) and the maximum likelihood scale estimator $\hat{\sigma}_n^2 = \hat{\sigma}_{\textup{ML},n}^2$ (right) for 50 samples from $X_0$ in the univariate Case 1 described in \ref{['sec:numerical-results']}.

Theorems & Definitions (40)

• theorem 2.1
• remark 2.2
• definition 2.3: Rate-unbiasedness
• Proposition 2.4
• definition 2.5: Sobolev kernel
• remark 2.6
• definition 3.1: Quasi-uniform sequence
• theorem 3.2
• theorem 3.3
• proof