Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference

TL;DR

This work tackles the challenge of learning and online inference in Gaussian Process State-Space Models under non-mean-field variational approximations. By embedding the Ensemble Kalman Filter into a variational framework (EnVI), it eliminates heavy variational parameterization, yielding a tractable ELBO computable from EnKF steps and differentiable for gradient-based optimization. The sparse GP extension and online variant (OEnVI) enable scalable training and streaming inference with principled objectives. Across synthetic and real datasets, EnVI and OEnVI consistently outperform MF/NMF baselines and online competitors, delivering accurate state estimation, robust dynamics learning, and efficient online operation. This approach provides a practical, principled alternative for nonparametric GPSSMs in online settings with strong uncertainty quantification.

Abstract

The Gaussian process state-space models (GPSSMs) represent a versatile class of data-driven nonlinear dynamical system models. However, the presence of numerous latent variables in GPSSM incurs unresolved issues for existing variational inference approaches, particularly under the more realistic non-mean-field (NMF) assumption, including extensive training effort, compromised inference accuracy, and infeasibility for online applications, among others. In this paper, we tackle these challenges by incorporating the ensemble Kalman filter (EnKF), a well-established model-based filtering technique, into the NMF variational inference framework to approximate the posterior distribution of the latent states. This novel marriage between EnKF and GPSSM not only eliminates the need for extensive parameterization in learning variational distributions, but also enables an interpretable, closed-form approximation of the evidence lower bound (ELBO). Moreover, owing to the streamlined parameterization via the EnKF, the new GPSSM model can be easily accommodated in online learning applications. We demonstrate that the resulting EnKF-aided online algorithm embodies a principled objective function by ensuring data-fitting accuracy while incorporating model regularizations to mitigate overfitting. We also provide detailed analysis and fresh insights for the proposed algorithms. Comprehensive evaluation across diverse real and synthetic datasets corroborates the superior learning and inference performance of our EnKF-aided variational inference algorithms compared to existing methods.

Figures (11)

• Figure 1: Graphical model of GPSSM. The white circles represent the latent variables, while the gray circles represent the observable variables. The thick horizontal bar represents a set of fully connected nodes, i.e., the GP.
• Figure 2: EnVI (top) & OEnVI (bottom) on state inference in linear Gaussian SSM. The RMSE of the latent state estimates for KF, EnVI, and OEnVI are 0.5252, 0.6841, and 0.7784, respectively; the RMSE between the observations and the latent states is 0.9872.
• Figure 3: Kink transition function learning performance (mean $\pm$$2 \sigma$) using various methods across different levels of emission noise ($\sigma_{\mathrm{R}}^2 \in \{0.008, 0.08, 0.8\}$, from left to right).
• Figure 4: Kink function learning performance against the training iterations. EnVI exhibits rapid convergence compared to vGPSSM and VCDT.
• Figure 5: Online NASCAR$^\circledR$ dynamics learning results of the three online algorithms. The prediction RMSE values for OEnVI, SVMC, and VJF are 1.8780, 4.6682, and 10.8499, respectively.

Theorems & Definitions (14)

• Remark 1
• Remark 2
• Theorem 1
• proof
• Proposition 1
• proof
• Remark 3
• Remark 4
• Proposition 2
• proof