Stationarity without mean reversion in improper Gaussian processes

TL;DR

The paper tackles the limitation that regular stationary GP covariances enforce mean reversion, which can lead to pathological extrapolation. It introduces improper GP priors with infinite variance and non-positive kernels to realize stationary but non-mean-reverting processes, deriving the analytic posterior via a simple correction and proposing a family of CPD kernels (including Smooth Walk, Gaussian Walk, and Matérn Walk) that approximate standard kernels. By combining theoretical development with extensive experiments on synthetic data, stock prices, and UCI datasets, it demonstrates reduced mean-reversion pathologies while retaining favorable kernel properties and predictive performance. The work lays a foundation for broader kernel design under improper priors, with implications for more calibrated, non-mean-reverting GP models in practice, and identifies avenues for rigorous high-dimensional theory and kernel construction.

Abstract

The behavior of a GP regression depends on the choice of covariance function. Stationary covariance functions are preferred in machine learning applications. However, (non-periodic) stationary covariance functions are always mean reverting and can therefore exhibit pathological behavior when applied to data that does not relax to a fixed global mean value. In this paper we show that it is possible to use improper GP priors with infinite variance to define processes that are stationary but not mean reverting. To this aim, we use of non-positive kernels that can only be defined in this limit regime. The resulting posterior distributions can be computed analytically and it involves a simple correction of the usual formulas. The main contribution of the paper is the introduction of a large family of smooth non-reverting covariance functions that closely resemble the kernels commonly used in the GP literature (e.g. squared exponential and Matérn class). By analyzing both synthetic and real data, we demonstrate that these non-positive kernels solve some known pathologies of mean reverting GP regression while retaining most of the favorable properties of ordinary smooth stationary kernels.

Figures (5)

• Figure 1: Samples and $95\%$ probability intervals of a) bilateral Brownian motion starting at $-1$ and b) superposition of two bilateral Brownian motions starting at $-1$ and $1$ respectively. Note the 'stationary' interval with constant variance in between the two points.
• Figure 2: Posterior expectation and $95\%$ intervals and samples of a GP conditioned on two data-points with A) improper Smooth Walk prior (top figure) and B) proper Squared Exponential prior. The length scale was $0.2$ for both models. The black line show the ground-truth signal $\text{sign}(t)$.
• Figure 3: Posterior expectation and $95\%$ interval of a GP regression with ground-truth signal $\sin(6 t) + 0.4 t - 5 \tanh(t)$ with noise level $\sigma = 0.05$. The length scale was selected by optimizing the marginal likelihood conditioned on one data point (see rightmost panel).
• Figure 4: Posterior expectation of a two-dimensional GP regression with ground-truth signal (test function) $\tanh(-x^2-y^2 + 5)$. The posterior expectation with the Squared Exponential prior reverses to the prior mean while the bivariate Smooth Walk expectation stays close to the nearest data-point.
• Figure 5: Numerical test of (improper) positive-definiteness of smooth conditionally positive definite kernels.

Theorems & Definitions (1)

• Definition 5.1: conditionally positive definite kernel