Modeling Information Blackouts in Missing Not-At-Random Time Series Data

TL;DR

This work tackles structured sensor blackouts in traffic time series by formulating a latent state-space model that explicitly treats missing data as MNAR through a state-dependent dropout channel. An approximate inference pipeline combining an MNAR-aware Extended Kalman Filter and RTS smoother, together with an EM-based training procedure and detector-specific updates, enables joint estimation of latent traffic dynamics and missingness mechanisms. Empirical results on Seattle loop data show that modeling latent dynamics yields substantial reconstruction and short-horizon forecasting gains, while explicit MNAR modeling provides consistent but modest improvements when dropout is informative, with stronger benefits in controlled MNAR settings. The findings highlight the central role of temporal dynamics and clarify when principled MNAR refinements are most valuable for sensor networks in traffic systems.

Abstract

Large-scale traffic forecasting relies on fixed sensor networks that often exhibit blackouts: contiguous intervals of missing measurements caused by detector or communication failures. These outages are typically handled under a Missing At Random (MAR) assumption, even though blackout events may correlate with unobserved traffic conditions (e.g., congestion or anomalous flow), motivating a Missing Not At Random (MNAR) treatment. We propose a latent state-space framework that jointly models (i) traffic dynamics via a linear dynamical system and (ii) sensor dropout via a Bernoulli observation channel whose probability depends on the latent traffic state. Inference uses an Extended Kalman Filter with Rauch-Tung-Striebel smoothing, and parameters are learned via an approximate EM procedure with a dedicated update for detector-specific missingness parameters. On the Seattle inductive loop detector data, introducing latent dynamics yields large gains over naive baselines, reducing blackout imputation RMSE from 7.02 (LOCF) and 5.02 (linear interpolation + seasonal naive) to 4.23 (MAR LDS), corresponding to about a 64% reduction in MSE relative to LOCF. Explicit MNAR modeling provides a consistent but smaller additional improvement on real data (imputation RMSE 4.20; 0.8% RMSE reduction relative to MAR), with similar modest gains for short-horizon post-blackout forecasts (evaluated at 1, 3, and 6 steps). In controlled synthetic experiments, the MNAR advantage increases as the true missingness dependence on latent state strengthens. Overall, temporal dynamics dominate performance, while MNAR modeling offers a principled refinement that becomes most valuable when missingness is genuinely informative.

Figures (7)

• Figure 1: Seattle Loop (2015) imputation RMSE on aligned validation windows with 95% bootstrap confidence intervals.
• Figure 2: Qualitative reconstructions on Seattle Loop. Shaded regions denote artificially masked blackout intervals. LOCF freezes at the last observed value, while LDS models infer a latent trajectory through the outage; MNAR and MAR are often close but MNAR can yield slightly sharper latent corrections when missingness correlates with the underlying state.
• Figure 3: Imputation RMSE by blackout length bucket (counts shown for the aligned evaluation pool). LOCF error grows rapidly with longer blackouts, while LDS-based imputations are substantially more stable.
• Figure 4: Seattle Loop forecasting RMSE vs horizon with 95% bootstrap confidence intervals. State-space modeling dominates the gain over heuristic baselines; MNAR provides a small, consistent refinement over MAR across horizons.
• Figure 5: Approximate-EM training objective (log-likelihood) for MAR and MNAR on Seattle Loop. In our runs it increases steadily and MNAR attains higher values.