Low-rank computation of the posterior mean in Multi-Output Gaussian Processes

TL;DR

The paper develops scalable posterior mean computation for multi-output Gaussian processes with separable Kronecker-structured covariances. By formulating the prediction as a Stein equation and employing low-rank Krylov methods (lrpcg) with KPIK preconditioning, it achieves efficient inference on large spatio-temporal graphs; it also introduces a degree-weighted average covariance for stationary-like data to further accelerate convergence. The work connects MOGPs to graph signal processing via graph filters, analyzes the role of eigendecompositions, and provides extensive numerical experiments on real road networks, demonstrating scalability to large graphs. Overall, it offers practical, scalable tools for MOGP regression on graphs and spatio-temporal data, with clear guidance on method choice and conditioning considerations.

Abstract

Gaussian processes (GP) are a versatile tool in machine learning and computational science. We here consider the case of multi-output Gaussian processes (MOGP) and present low-rank approaches for efficiently computing the posterior mean of a MOGP. Starting from low-rank spatio-temporal data we consider a structured covariance function, assuming separability across space and time. This separability, in turn, gives a decomposition of the covariance matrix into a Kronecker product of individual covariance matrices. Incorporating the typical noise term to the model then requires the solution of a large-scale Stein equation for computing the posterior mean. For this, we propose efficient low-rank methods based on a combination of a LRPCG method with the Sylvester equation solver KPIK adjusted for solving Stein equations. We test the developed method on real world street network graphs by using graph filters as covariance matrices. Moreover, we propose a degree-weighted average covariance matrix, which can be employed under specific assumptions to achieve more efficient convergence.

Figures (6)

• Figure 1: Perspectives in space and time for graph-based data. \newlabelfig:graphs0
• Figure 1: Degree-Weighted Average: $v_1$ distributes $\frac{1}{2}$ and $v_2$ distributes $\frac{1}{5}$ of an abstract quantity to the (unknown) $v^\ast$. The proportions are assumed to be equal to all neighbors and therefore $1/\text{degree}$.
• Figure 1: Oxford street network example: 30% of randomly chosen input nodes and 70% output nodes (red).
• Figure 2: Regression on data from dynamical system
• Figure 3: Regression on data following stationary property