Beyond State Space Representation: A General Theory for Kernel Packets

TL;DR

The paper introduces kernel packets (KPs), a general framework that extends state-space GP inference beyond one-dimensional grids to multi-dimensional scattered data while preserving exactness and linear-time training. By coupling forward and backward SS representations, KPs yield compactly supported basis functions that sparsify the kernel matrix and enable fast posterior mean, variance, and log-likelihood computations. It provides a complete algorithm to construct KP bases for general kernels, including additive and product forms, and extends to multidimensional settings via tensor and Kronecker structures. Numerical experiments on millions of samples show exact, memory-efficient inference where SS-based and low-rank methods fail, with applications to additive and product-form GPs in real-time MRI and fluid dynamics. Overall, KPs bridge SDE-based and kernel-based GP inference, offering scalable, exact GP regression for large, multidimensional datasets.

Abstract

Gaussian process (GP) regression provides a flexible, nonparametric framework for probabilistic modeling, yet remains computationally demanding in large-scale applications. For one-dimensional data, state space (SS) models achieve linear-time inference by reformulating GPs as stochastic differential equations (SDEs). However, SS approaches are confined to gridded inputs and cannot handle multi-dimensional scattered data. We propose a new framework based on kernel packet (KP), which overcomes these limitations while retaining exactness and scalability. A KP is a compactly supported function defined as a linear combination of the GP covariance functions. In this article, we prove that KPs can be identified via the forward and backward SS representations. We also show that the KP approach enables exact inference with linear-time training and logarithmic or constant-time prediction, and extends naturally to multi-dimensional gridded or scattered data without low-rank approximations. Numerical experiments on large-scale additive and product-form GPs with millions of samples demonstrate that KPs achieve exact, memory-efficient inference where SDE-based and low-rank GP methods fail.

Figures (7)

• Figure 1: KPs corresponding to Matérn-$3/2$ and $5/2$ correlations from chen2022kernel. KPs, left-sided KPs, and right-sided KPs are plotted in orange, blue, and green lines, respectively.
• Figure 2: First column: KP is linear combinations of seven combined kernels; Middle column: Twenty kernel functions at $\{t_i=1+i/10\}_{i=1}^{20}$ that forms function spaces $\{K(\cdot,t_i)\}$; Last column: KP basis associated to combined kernel.
• Figure 3: Two-dimensional KP of additive (left) and product (right) Matérn kernels
• Figure 4: $\mathbf H$ is a block-tridiagonal matrix. When working on the $j$-th column, we can get $\mathbf M_{j}^-=\mathbf M_{j-1}^+$ directly by symmetry and solve an auxiliary matrix $\mathbf M_j^{--}$ by putting $[\mathbf M_{j-2};\mathbf M_{j-1}^-;\mathbf M_j^{--}]$ in a consecutive column (left); then we use $[\mathbf M_j^{--}, \mathbf M_j^- ,\mathbf M_j]$ to solve $\mathbf M_j$ (middle), and $[\mathbf M_j^{-}, \mathbf M_j ,\mathbf M_j^+]$ to solve $\mathbf M_j^+$ (right).
• Figure 5: Left: Ten groups of points in different colors are selected from 1,000 sample points to construct ten KP functions. Right: The plots of the ten KP functions for product kernel $K(\mathbf t,\mathbf t')=e^{-|t_1-t_1'|-|t_2-t_2'|}$ corresponding to the selected groups of points.

Theorems & Definitions (29)

• definition 1
• definition 2
• theorem 1: chen2022kernel
• theorem 2
• proof
• theorem 3
• theorem 4: Main Theorem
• remark 1
• theorem 5
• theorem 6