Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

Zachariah Malik, Romit Maulik

TL;DR

This work studies two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one.

Abstract

Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.

A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering

TL;DR

This work studies two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one.

Abstract

Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.
Paper Structure (5 sections, 26 equations, 4 figures, 3 tables, 3 algorithms)

This paper contains 5 sections, 26 equations, 4 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Ensemble Kalman Filtering
  3. Score Filter
  4. Experimental Results
  5. Conclusion

Figures (4)

  • Figure 1: Assimilated trajectories (red) against true trajectory (black) and cubic perturbed observations (blue stars) for SF (left), CG-EnKF (middle), and NS-EnKF (right), for L96 along dimensions $1$ (top), $20$ (middle), and $40$ (bottom).
  • Figure 2: Histograms of the analysis ensemble of the first dimension at the $30$th (top) time step, the $15$th dimension at the $90$th time step (middle), and the $40$th dimension as the $50$th time step (bottom) for each of the models, with cubic perturbations. The blue line gives the mean of the ensemble and the red line gives the ground truth.
  • Figure 3: NS-EnKF assimilation of the first $100$ assimilation cycles of the first dimension of L-96 with Pareto additive noise.
  • Figure 4: Assimilated trajectories (red) against true trajectory (black) and cubic perturbed observations (blue stars) for vanilla EnKF (left), CG-EnKF (middle), and NS-EnKF (right), for the first dimension of L96 with Gaussian additive noise (top), bimodal additive noise (middle), and exponential additive noise (bottom).