An Adaptive Online Smoother with Closed-Form Solutions and Information-Theoretic Lag Selection for Conditional Gaussian Nonlinear Systems

TL;DR

The paper tackles state estimation for complex turbulent systems by developing an adaptive-online smoother within the conditional Gaussian nonlinear system (CGNS) framework, achieving closed-form forward–backward smoothing updates with no empirical tuning. An information-theoretic criterion is introduced to determine adaptive lags, balancing posterior uncertainty reduction against storage and computation, and enabling lag adjustments that respond to intermittency and extreme events. The methodology yields exact Gaussian posteriors for both filtering and smoothing, along with discrete-time online updates and a fixed-lag baseline for comparison, and is supported by theoretical analysis of update matrices and spectral properties. The adaptive-lag approach is demonstrated through three applications: online causality detection in a nonlinear dyad, high-dimensional Lagrangian data assimilation, and online parameter estimation with partial observations, including an online EM algorithm that benefits from the smoother’s analytic formulas. Collectively, the work offers a principled, scalable, and storage-efficient framework for real-time state estimation, causal inference, and online learning in CGNS-driven problems with strong nonlinear and non-Gaussian features.

Abstract

Data assimilation (DA) combines partial observations with dynamical models to improve state estimation. Filter-based DA uses only past and present data and is the prerequisite for real-time forecasts. Smoother-based DA exploits both past and future observations. It aims to fill in missing data, provide more accurate estimations, and develop high-quality datasets. However, the standard smoothing procedure requires using all historical state estimations, which is storage-demanding, especially for high-dimensional systems. This paper develops an adaptive-lag online smoother for a large class of complex dynamical systems with strong nonlinear and non-Gaussian features, which has important applications to many real-world problems. The adaptive lag allows the utilization of observations only within a nearby window, thus reducing computational complexity and storage needs. Online lag adjustment is essential for tackling turbulent systems, where temporal autocorrelation varies significantly over time due to intermittency, extreme events, and nonlinearity. Based on the uncertainty reduction in the estimated state, an information criterion is developed to systematically determine the adaptive lag. Notably, the mathematical structure of these systems facilitates the use of closed analytic formulae to calculate the online smoother and adaptive lag, avoiding empirical tunings as in ensemble-based DA methods. The adaptive online smoother is applied to studying three important scientific problems. First, it helps detect online causal relationships between state variables. Second, the advantage of reduced computational storage expenditure is illustrated via Lagrangian DA, a high-dimensional nonlinear problem. Finally, the adaptive smoother advances online parameter estimation with partial observations, emphasizing the role of the observed extreme events in accelerating convergence.

Figures (9)

• Figure 3.1: Schematic diagram of how the online discrete smoother update works. Only the smoother mean $\boldsymbol{\mu}_{\text{s}}^{\cdot,\cdot}$ is depicted; the same diagram applies to the smoother covariance $\mathbf{R}_{\text{s}}^{\cdot,\cdot}$ as well, without loss of generality.
• Figure 3.1: Panels (a)--(f): Same as Panels (a)--(f) of Figure \ref{['fig:LDA_Ice_Floes_Fig_2']}, but for the dyad-interaction model, \ref{['eq:dyad1']}--\ref{['eq:dyad2']}, from the case study in Section \ref{['sec:4.1']}. Panel (g): Same as Panel (e) but using the LSDev sequence to define the adaptive lags (per \ref{['eq:adaptlaginfodef1']}--\ref{['eq:adaptlaginfodef2']}).
• Figure 4.1: Panel (a): Trajectory of the observable variable $u$. Panel (b): True trajectory of the unobserved variable $v$ (in blue), alongside the filter (in green) and offline smoother (in red) posterior mean time series. Their respective first two standard deviations away from the mean state are also plotted through correspondingly colored shaded regions, with the standard deviation in this case being exactly equal to the square root of the respective posterior covariance. The threshold above which $v$ acts as anti-damping to $u$, i.e., the line $y=d_u/\gamma=1/6$, is also plotted in a dashed black line. Panel (c): Temporal evolution of the information gain of the filter and smoother posterior Gaussian statistics beyond the statistical attractor of the unobserved variable. This is calculated using the signal--dispersion decomposition of the relative entropy in \ref{['eq:signaldispersion']}. A logarithmic scale is used for the y-axis. Panel (d): Time-averaged PDF of $u$ calculated using the observations over $t\in[20,60]$ (in black), alongside its Gaussian fit density defined by the mean and standard deviation of the signal in the same time period (in dashed magenta). Panel (e): Same as (d) but for the unobserved variable $v$, together with the PDFs corresponding to the filter and smoother posterior mean time series. Panel (f): PDF of the unobserved variable's prior statistical attractor, $p_{\text{att}}(v)$.
• Figure 4.2: Panels (a)--(f): Real-time comparison between the filter posterior mean time series (in green) and the one generated from the adaptive-lag online smoother strategy by Theorem \ref{['thm:onlinesmoother']} (in purple). The observed trajectory of $u$ is plotted in Panel (a) (in black), while the true trajectory of $v$ is showed in all panels for reference (in blue). Panel (g): Adaptive lag values (measured in time units, i.e., $L_n\Delta t$) generated by the algorithm defined in Section \ref{['sec:3.4.2']}, specifically through \ref{['eq:adaptlaginfodeforiginalseq']} and \ref{['eq:adaptlaginfodef2']} (i.e., using the original sequence of relative entropies). Panel (h): Standardized information gain criterion in \ref{['eq:infogainstandardized']} as a function of $n$ (for $n\Delta t\in[32.5,40]$) and of $n\Delta t-j\Delta t$ (for $j\Delta t\in[R_n\Delta t,n\Delta t]$). Plotted on a logarithmically scaled colormap. Panel (i): Same as Panel (h) but for the spectral radii of the update values $D^{j,n-2}$, i.e., $|D^{j,n-2}|$. Panels (h) and (i) share the same colorbar, and have their y-axis flipped.
• Figure 4.3: Panel (a): True state of the underlying flow field at $t=2$, where the colormap shows the amplitude and the quiver plot represents the velocity field. It is superimposed by the location of the tracers and their velocity vectors. The magnitudes of the quiver plot for the ocean's velocity field and velocity vectors for the tracers are on distinct scales. Panels (b)--(c): Similar to (a), but corresponding to the estimated state from the posterior mean, calculated through the filter and smoother, respectively. Panels (d)--(e): Comparison between the true time series (in blue) and the posterior mean time series of the filter (in green) and smoother (in red) solutions for the real part of the ocean Fourier mode $\mathbf{k}=(2,-1)^\mathtt{T}$ and zonal velocity of tracer #1 (labeled in Panels (a)--(c)).

Theorems & Definitions (8)

• Theorem 2.1: Conditional Gaussianity
• proof : Proof
• Theorem 2.2: Optimal Nonlinear Filter State Estimation Equations
• proof : Proof
• Theorem 2.3: Optimal Nonlinear Smoother State Estimation Backward Equations
• proof : Proof
• Theorem 3.1: Optimal Online Forward-In-Time Discrete Smoother
• proof : Proof of Theorem \ref{['thm:onlinesmoother']}