An Ensemble Information Filter: Retrieving Markov-information from the SPDE discretisation

TL;DR

This work tackles spurious correlations and ensemble collapse in high-dimensional ensemble data assimilation by embedding Markov locality directly into the statistical model. It introduces the Ensemble Information Filter (EnIF), which encodes locality via sparse precision matrices derived from SPDE operators and learns a Gaussian prior $Q$ through affine triangular (KR) measure transport, removing the need for ad-hoc localisation. The method updates ensembles in the information (canonical) space using Kalman-like information filters, with conditional independence structured by a graph $oldsymbol{ ext{G}}$, enabling scalable, adaptive, and locality-respecting data assimilation across smoothing, filtering, and parameter estimation. Through synthetic and applied experiments, EnIF demonstrates improved statistical consistency, automatic localisation, and robustness to varying dependence strengths, offering a principled alternative to EnKF/ES that scales to large SPDE-driven systems. The approach hinges on combining SPDE-informed Markov structure, affine KR-maps for $Q$, and sparse/efficient precision-based updates to deliver accurate, locality-aware assimilation without manual tuning.

Abstract

Ensemble-based Data Assimilation faces significant challenges in high-dimensional systems due to spurious correlations and ensemble collapse. These issues arise from estimating dense dependencies with limited ensemble sizes. This paper introduces the Ensemble Information Filter, which encodes Markov properties directly into the statistical model's precision matrix, leveraging structure from SPDE dynamics to constrain information to propagate locally. EnIF eliminates the need for ad-hoc localisation, improving statistical consistency and scalability. Numerical experiments demonstrate its advantages in filtering, smoothing, and parameter estimation, making EnIF a robust and efficient solution for large-scale data assimilation problems.

Figures (10)

• Figure 1: Left: 50 realisations of 1-d Matérn ($\kappa=0.1$) Gaussian process as in Example \ref{['ex:conditional-matern-exact']} sampled on $x\in[0,1]$, unconditioned on an observation of the endpoint (green dot) at $x=1.0$. Right: Conditioned realisations overlayd the unconditional ones from the left plot. The update is a special case of the EnKF and is expressed in Example \ref{['ex:conditional-matern-exact']}. Red line showcases the first Matérn realisation (left) and how this realisation is updated (orange, right).
• Figure 2: Left: Conditioned realisations as in Figure \ref{['fig:matern-exact']} right, but exchanging true covariance with sample covariance using the 50 unconditioned Matérn realisations. Right: 30 different sample correlations with the endpoint $x=1.0$ based on $n=50$ realisations, along with the true population correlation. The green line is the exact sample-correlations used for the update in the left plot.
• Figure 3: Average KLD values \ref{['eq:kullback-leibler-gaussian']} for the EnIF (orange), EnKF/ES (green), and the Euler-Maruyama scheme (blue) relative to the true analytical posterior of the 1D Matérn process ($\kappa=1.0$) corresponding to the Ornstein-Uhlenbeck process \ref{['eq:ornstein-uhlenbeck-matern-sde']}. The $x$-axis shows the level of resolution, indicating the number of discretisation points and thus the number of vertices.
• Figure 4: Exact, ES and EnIF updates of the first realisation in an $n=50$ ensemble of AR-1 processes with varying dependence strength, $\phi \in [0.0, 0.5, 0.9, 0.95]$.
• Figure 5: KLD values \ref{['eq:kullback-leibler-gaussian']} vs. localisation radius of ES/EnKF. Both the $x$- and $y$-axes are on a log-scale.

Theorems & Definitions (5)

• Example 1: Conditioned Matérn-GRF
• Example 2: Affine Triangular Measure Transport
• Example 3: Ensemble Smoother on Matérn
• Example 4: Markov solutions to the Matérn SPDE
• Example 5: FEM solution to Matérn ($\alpha=2$)