Latent-EnSF: A Latent Ensemble Score Filter for High-Dimensional Data Assimilation with Sparse Observation Data

TL;DR

A novel data assimilation method is proposed, Latent-EnSF, which leverages EnSF with efficient and consistent latent representations of the full states and sparse observations to address the joint challenges of high dimensionlity in states and high sparsity in observations for nonlinear Bayesian filtering.

Abstract

Accurate modeling and prediction of complex physical systems often rely on data assimilation techniques to correct errors inherent in model simulations. Traditional methods like the Ensemble Kalman Filter (EnKF) and its variants as well as the recently developed Ensemble Score Filters (EnSF) face significant challenges when dealing with high-dimensional and nonlinear Bayesian filtering problems with sparse observations, which are ubiquitous in real-world applications. In this paper, we propose a novel data assimilation method, Latent-EnSF, which leverages EnSF with efficient and consistent latent representations of the full states and sparse observations to address the joint challenges of high dimensionlity in states and high sparsity in observations for nonlinear Bayesian filtering. We introduce a coupled Variational Autoencoder (VAE) with two encoders to encode the full states and sparse observations in a consistent way guaranteed by a latent distribution matching and regularization as well as a consistent state reconstruction. With comparison to several methods, we demonstrate the higher accuracy, faster convergence, and higher efficiency of Latent-EnSF for two challenging applications with complex models in shallow water wave propagation and medium-range weather forecasting, for highly sparse observations in both space and time.

Figures (10)

• Figure 1: Flow of the Latent-EnSF. An ensemble of prior states $x_{t}|y_{1:t-1}$ is assimilated with sparse observation $y_{t}$ in the latent autoencoder space to obtain samples $x_{t}|y_{1:t}$ from the posterior.
• Figure 2: The gradient of the log-likelihood function $\nabla_x P(y_t|x_t)$ (right) vanishes at the points where the sparse observation data $y_t$ (middle) do not have any information of the state $x_t$ (left).
• Figure 3: A coupled VAE for consistent latent representations of sparse observations and full states.
• Figure 4: Evolution of states and sparse observations.
• Figure 5: Relative RMSE of EnSF and Latent-EnSF for sparse observations.