Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Deep Attention Recurrent Neural Network for Heterogeneous Time Series Imputation

Linglong Qian, Zina Ibrahim, Richard Dobson

TL;DR

DEARI addresses non-random missingness in heterogeneous multivariate time series by unifying deep attention-based imputation with self-supervised metric learning and Bayesian marginalization. By extending the BRITS backbone with self-attention, residuals, and a deep metric objective, it enables scalable deep architectures that improve imputation accuracy while providing explicit uncertainty estimates. Empirical results across air quality, healthcare, and traffic datasets show deterministic and Bayesian DEARI surpass state-of-the-art baselines, with uncertainty quantification aiding reliability. This approach offers practical benefits for time-series analysis in domains with complex missingness, facilitating both accurate imputations and trustworthy confidence bounds for downstream decisions.

Abstract

Missingness is ubiquitous in multivariate time series and poses an obstacle to reliable downstream analysis. Although recurrent network imputation achieved the SOTA, existing models do not scale to deep architectures that can potentially alleviate issues arising in complex data. Moreover, imputation carries the risk of biased estimations of the ground truth. Yet, confidence in the imputed values is always unmeasured or computed post hoc from model output. We propose DEep Attention Recurrent Imputation (DEARI), which jointly estimates missing values and their associated uncertainty in heterogeneous multivariate time series. By jointly representing feature-wise correlations and temporal dynamics, we adopt a self attention mechanism, along with an effective residual component, to achieve a deep recurrent neural network with good imputation performance and stable convergence. We also leverage self-supervised metric learning to boost performance by optimizing sample similarity. Finally, we transform DEARI into a Bayesian neural network through a novel Bayesian marginalization strategy to produce stochastic DEARI, which outperforms its deterministic equivalent. Experiments show that DEARI surpasses the SOTA in diverse imputation tasks using real-world datasets, namely air quality control, healthcare and traffic.

Uncertainty-Aware Deep Attention Recurrent Neural Network for Heterogeneous Time Series Imputation

TL;DR

DEARI addresses non-random missingness in heterogeneous multivariate time series by unifying deep attention-based imputation with self-supervised metric learning and Bayesian marginalization. By extending the BRITS backbone with self-attention, residuals, and a deep metric objective, it enables scalable deep architectures that improve imputation accuracy while providing explicit uncertainty estimates. Empirical results across air quality, healthcare, and traffic datasets show deterministic and Bayesian DEARI surpass state-of-the-art baselines, with uncertainty quantification aiding reliability. This approach offers practical benefits for time-series analysis in domains with complex missingness, facilitating both accurate imputations and trustworthy confidence bounds for downstream decisions.

Abstract

Missingness is ubiquitous in multivariate time series and poses an obstacle to reliable downstream analysis. Although recurrent network imputation achieved the SOTA, existing models do not scale to deep architectures that can potentially alleviate issues arising in complex data. Moreover, imputation carries the risk of biased estimations of the ground truth. Yet, confidence in the imputed values is always unmeasured or computed post hoc from model output. We propose DEep Attention Recurrent Imputation (DEARI), which jointly estimates missing values and their associated uncertainty in heterogeneous multivariate time series. By jointly representing feature-wise correlations and temporal dynamics, we adopt a self attention mechanism, along with an effective residual component, to achieve a deep recurrent neural network with good imputation performance and stable convergence. We also leverage self-supervised metric learning to boost performance by optimizing sample similarity. Finally, we transform DEARI into a Bayesian neural network through a novel Bayesian marginalization strategy to produce stochastic DEARI, which outperforms its deterministic equivalent. Experiments show that DEARI surpasses the SOTA in diverse imputation tasks using real-world datasets, namely air quality control, healthcare and traffic.
Paper Structure (28 sections, 16 equations, 3 figures, 1 table)

This paper contains 28 sections, 16 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Terminology and Background
  4. Overview of the BRITS Backbone
  5. Methodology
  6. Deep Attention Recurrent neural network
  7. Self-Attention Initialization
  8. Deep Self-Supervised Metric Learning
  9. Online Hard Triplets Mining
  10. Multilayer Adaptation
  11. Loss Function
  12. Bayesian Marginalization Strategy
  13. Loss Function
  14. Experiments
  15. Datasets
  16. ...and 13 more sections

Figures (3)

  • Figure 1: An example of multivariate time-series. $x_{1-5}$: observations in time steps $t_1,\,...,\,t_5$ with corresponding time-stamps $s_{1-5}=0,\,4,\,5,\,7,\,9$. Feature $d_2$ was missing during $t_{2-4}$, the last observation took place at $s_1$. Hence, $\delta_5^2 =t_5-t_1 =9-0=9$.
  • Figure 2: The structure of the proposed methods. a) deep-attention recurrent neural network, which collapses to BRITS if No. layers = 1. b) deep self-supervised metric learning, larger batch sizes lead to more triples. c) Bayesian marginalization, all parameters of the Bayesian layers are controlled by distributions.
  • Figure 3: Ablation Studies: B: BRITS; D: DEARI; digit: Number of encoder layers; one/more: Triple Size. In (d): C: Model complexity (number of parameters); P: Performance (MAE loss)