Latent Autoencoder Ensemble Kalman Filter for Data assimilation

TL;DR

A latent autoencoder ensemble Kalman filter (LAE-EnKF) that addresses this limitation by reformulating the assimilation problem in a learned latent space with linear and stable dynamics and yields more accurate and stable assimilation than the standard EnKF and related latent-space methods.

Abstract

The ensemble Kalman filter (EnKF) is widely used for data assimilation in high-dimensional systems, but its performance often deteriorates for strongly nonlinear dynamics due to the structural mismatch between the Kalman update and the underlying system behavior. In this work, we propose a latent autoencoder ensemble Kalman filter (LAE-EnKF) that addresses this limitation by reformulating the assimilation problem in a learned latent space with linear and stable dynamics. The proposed method learns a nonlinear encoder--decoder together with a stable linear latent evolution operator and a consistent latent observation mapping, yielding a closed linear state-space model in the latent coordinates. This construction restores compatibility with the Kalman filtering framework and allows both forecast and analysis steps to be carried out entirely in the latent space. Compared with existing autoencoder-based and latent assimilation approaches that rely on unconstrained nonlinear latent dynamics, the proposed formulation emphasizes structural consistency, stability, and interpretability. We provide a theoretical analysis of learning linear dynamics on low-dimensional manifolds and establish generalization error bounds for the proposed latent model. Numerical experiments on representative nonlinear and chaotic systems demonstrate that the LAE-EnKF yields more accurate and stable assimilation than the standard EnKF and related latent-space methods, while maintaining comparable computational cost and data-driven.

Figures (14)

• Figure 1: The architecture of latent autoencoder framework. It consists of four principal trainable components: a state encoder $\mathscr{E}$ that projects physical states into a compact latent space, a decoder $\mathscr{D}$ that reconstructs physical states from latent representations, a linear transition operator $\mathbf{A}$ that governs latent dynamics, and an observation encoder $\mathscr{E}_{\text{obs}}$ that maps observations into the same latent manifold.
• Figure 1: Learned latent representations for latent dimensions $n=2,3,4$. Top: latent variables learned by the autoencoder component of DAE-EnKF (denoted as DAE), where no structural constraint is imposed on the latent dynamics. Bottom: latent variables learned by the proposed LAE, which enforces linear evolution in the latent space. For $n>2$, the latent variables are projected onto their first two principal components using PCA for visualization.
• Figure 2: Pairwise latent coordinate plots $(z_i, z_j)$ for $n=3$ learned by the LAE.
• Figure 3: Long-term prediction relative error for latent dimensions $n=2,3,4$. Predictions are obtained by recursively propagating the learned latent dynamics forward in time and decoding back to the physical space.
• Figure 4: Time evolution of the relative RMSE for different methods with latent dimension $n=2$. Solid curves show the mean over 10 independent runs, and shaded regions indicate the corresponding 95% confidence intervals.

Theorems & Definitions (7)

• Theorem 4.1
• Theorem 4.2
• Definition A.1: Covering number Vaart1996cover
• Lemma A.2: Chen2022cover
• Lemma A.3
• Proof 1
• Proof 2: Proof of Theorem \ref{['thm1']}