LD-EnSF: Synergizing Latent Dynamics with Ensemble Score Filters for Fast Data Assimilation with Sparse Observations

TL;DR

LD-EnSF tackles real-time data assimilation for high-dimensional, nonlinear systems with sparse observations by learning latent dynamics with LDNet and using an LSTM-based observation encoder. Assimilation is performed entirely in a compact latent space via an Ensemble Score Filter, avoiding costly full-space simulations and enabling fast updates. Across shallow water and Kolmogorov flow experiments, LD-EnSF achieves higher accuracy than EnSF and Latent-EnSF under sparse/noisy observations while delivering massive computational speedups (latent dimension ~$10$–$12$ vs full-space dimensions). This approach significantly improves the practicality of data assimilation in real-time, large-scale applications and opens avenues for further theoretical and architectural enhancements.

Abstract

Data assimilation techniques are crucial for correcting the trajectory when modeling complex physical systems. A recently developed data assimilation method, Latent Ensemble Score Filter (Latent-EnSF), has shown great promise in addressing the key limitation of EnSF for highly sparse observations in high-dimensional and nonlinear data assimilation problems. It performs data assimilation in a latent space for encoded states and observations in every assimilation step, and requires costly full dynamics to be evolved in the original space. In this paper, we introduce Latent Dynamics EnSF (LD-EnSF), a novel methodology that completely avoids the full dynamics evolution and significantly accelerates the data assimilation process, which is especially valuable for complex dynamical problems that require fast data assimilation in real time. To accomplish this, we introduce a novel variant of Latent Dynamics Networks (LDNets) to effectively capture and preserve the system's dynamics within a very low-dimensional latent space. Additionally, we propose a new method for encoding sparse observations into the latent space using Long Short-Term Memory (LSTM) networks, which leverage not only the current step's observations, as in Latent-EnSF, but also all previous steps, thereby improving the accuracy and robustness of the observation encoding. We demonstrate the robustness, accuracy, and efficiency of the proposed method for two challenging dynamical systems with highly sparse (in both space and time) and noisy observations.

Figures (6)

• Figure 1: The pipeline of the LD-EnSF method.Offline learning: In phase 1, the LDNet is trained based on the dataset to capture the latent dynamics. In phase 2, an LSTM encoder is trained to align the observation history $y_{1:t}$ with latent variables $s_t$ and parameters $u_t$. Online deployment: for each assimilation time step, the LD-EnSF assimilates an ensemble of prior latent pairs $\{s_{t},u_{t}\}$ with LSTM encoded latent pairs $(\hat{s}_{t},\hat{u}_{t})$. The posterior latent states can then be used to reconstruct the assimilated full states at arbitrary space and temporal points.
• Figure 2: Comparison of the reconstruction errors (left) of the VAE and LDNets (with and without retraining of the reconstruction network) for the shallow water equations (top) and Kolmogorov flow (bottom). The latent states of LDNets (middle) are much smoother than those of VAE (right).
• Figure 3: Visualization of the ground truth of the surface elevation $\eta$ of the shallow water dynamics \ref{['eq:shallow-water']} (1st row), LDNet predictions from known initial condition (2nd row), and the prediction errors (3rd row) at five time steps. Sparse observations at $10\times 10$ from $150\times 150$ spatial grid points (4th row) are assimilated to an ensemble of 20 samples of the LDNet dynamics starting from 20 random initial conditions by the LD-EnSF algorithm, which leads to the full states (5th row) reconstructed from one sample of the assimilated latent states, with assimilation errors (last row).
• Figure 4: Visualization of the vorticity field $\omega=\nabla\times\mathbf{v}$ of the ground truth of the Kolmogorov flow \ref{['eq:Kolmogorov-flow']} at Reynolds number $Re=1469.5$ (1st row) and the perturbed dynamics at $Re=545.2$ (2nd row), with their difference shown in the 3rd row. Sparse observations (4th row) at $10\times 10$ from $150\times 150$ spatial grid points are assimilated to an ensemble of 20 samples of the LDNet dynamics starting from 20 random samples of the Reynolds number, which leads to the full states (5th row) reconstructed from one sample of the assimilated latent states, with assimilation errors (last row).
• Figure 5: The relative RMSE of the assimilated latent states (left) and parameters (middle) compared to the latent states at the truth parameters, as well as the reconstructed full states (right) at different time steps and observation noises for the shallow water equations (top) and Kolmogorov flow (bottom). The errors of the unassimilated/original quantities are also shown in the plots.