Importance sampling for rare event tracking within the ensemble Kalman filtering framework

TL;DR

The paper tackles the challenge of estimating very small rare-event probabilities for the running maximum of a projected SDE state within the Ensemble Kalman Filter. It develops three IS strategies—biasing the initial condition, biasing the Wiener path via stochastic optimal control, and a joint bias—each grounded in solving a Kolmogorov backward equation with appropriate boundary conditions and made tractable in high dimensions through Markovian projection. Across three model problems (Double-Well SDE, Langevin dynamics, and a noisy Charney–deVore model), the proposed methods achieve substantial variance reduction compared with crude Monte Carlo and multilevel cross-entropy, enabling more reliable real-time rare-event tracking in EnKF. The contributions include PDE- and CE-based initial-density IS, SOC-based pathwise IS, and their combination, all integrated with adaptive time-stepping and Brownian-bridge corrections to handle boundary-crossing events efficiently, with broad applicability to high-dimensional data assimilation tasks.

Abstract

In this work we employ importance sampling (IS) techniques to track a small over-threshold probability of a running maximum associated with the solution of a stochastic differential equation (SDE) within the framework of ensemble Kalman filtering (EnKF). Between two observation times of the EnKF, we propose to use IS with respect to the initial condition of the SDE, IS with respect to the Wiener process via a stochastic optimal control formulation, and combined IS with respect to both initial condition and Wiener process. Both IS strategies require the approximation of the solution of Kolmogorov Backward equation (KBE) with boundary conditions. In multidimensional settings, we employ a Markovian projection dimension reduction technique to obtain an approximation of the solution of the KBE by just solving a one dimensional PDE. The proposed ideas are tested on three illustrative examples: Double Well SDE, Langevin dynamics and noisy Charney-deVore model, and showcase a significant variance reduction compared to the standard Monte Carlo method and another sampling-based IS technique, namely, multilevel cross entropy.

Figures (13)

• Figure 1: Illustration of the IS idea for the rare event tracking in the context of the EnKF.
• Figure 2: Illustration of the challenge with stopped diffusions in the boundary discussed in Remark \ref{['remark:brownianbridge']}. The continuous path (red) hits the threshold at $\tau_\mathcal{K}$ although the discrete EM solutions do not exit the boundary (or may exit after $\tau_\mathcal{K}$).
• Figure 3: A trajectory of the DW SDE from Section \ref{['ssec:dw']} over $T=100$ observation times with given observations and the double-well potential. The model parameters: $b = 0.5$, $u_0 \sim N(-0.7,0.1)$. The EnKF parameters: $H=1$, $\Gamma=0.1$.
• Figure 4: Double Well example. The original initial density ($\rho_{u_0}\sim N(-1, \sigma_0)$ in red solid line) is centered in one well with the threshold $\mathcal{K}$ in the other well. For all three values of $\sigma_0$, $\mathcal{K}$ is such that the rare event probability is around $10^{-3}$. A comparision of the approximated optimal initial densities based on: IS wrt $\rho_0$ ($\tilde{\rho}_{u_0}^{PDE,1}$ in cyan dash-dotted line), IS wrt both $\rho_0$ and $W(t)$ ($\tilde{\rho}_{u_0}^{PDE,2}$ in blue dash-dotted line), and CE-based IS ($\tilde{\rho}_{u_0}^{CE}$ in green dashed line).
• Figure 5: Double Well example Model parameter: $b=0.5$. 95% CI of the estimator with the simulation parameters: $T=1$, $\Delta t= 0.01$, $\mathcal{K}=1$, $u_0 \sim N(\mu_0,\sigma_0)$ with $\mu_0=-1$, $\sigma_0=0.5$.

Theorems & Definitions (4)

• Remark 1: Tail assumptions
• Remark 2: Stopped diffusions
• Example 1
• Remark 3: Alternative to Gaussian fitting