Irregularity-Informed Time Series Analysis: Adaptive Modelling of Spatial and Temporal Dynamics

TL;DR

A novel transformer-based framework for general irregular time series data is proposed that treats IRTS from four views: Locality, Time, Spatio and Irregularity to motivate the data usage to the highest potential and improves the generalization ability to various IRTS data.

Abstract

Irregular Time Series Data (IRTS) has shown increasing prevalence in real-world applications. We observed that IRTS can be divided into two specialized types: Natural Irregular Time Series (NIRTS) and Accidental Irregular Time Series (AIRTS). Various existing methods either ignore the impacts of irregular patterns or statically learn the irregular dynamics of NIRTS and AIRTS data and suffer from limited data availability due to the sparsity of IRTS. We proposed a novel transformer-based framework for general irregular time series data that treats IRTS from four views: Locality, Time, Spatio and Irregularity to motivate the data usage to the highest potential. Moreover, we design a sophisticated irregularity-gate mechanism to adaptively select task-relevant information from irregularity, which improves the generalization ability to various IRTS data. We implement extensive experiments to demonstrate the resistance of our work to three highly missing ratio datasets (88.4\%, 94.9\%, 60\% missing value) and investigate the significance of the irregularity information for both NIRTS and AIRTS by additional ablation study. We release our implementation in https://github.com/IcurasLW/MTSFormer-Irregular_Time_Series.git

Figures (5)

• Figure 1: (a) AIRTS irregularity pattern from PAM dataset (b)NIRTS irregularity pattern from P19 dataset
• Figure 2: MTSFormer Overall Architecture: (1) MTSFormer with $\mathcal{K}$ layer MTSFormer block accepts three different views as inputs: Local, Time and Sensor. Local, Time and Sensor views are encoded by three individual encoders $L^k_{tc}$, $L^k_{tt}$ and $L^k_{ts}$. (2) Sensor as token path transposes the feature dimension of $x^t_i$ to the reading of time dimension, focusing on the interaction between sensors. The encoder of of Sensor as token path is a one-layer vanilla transformer encoder. (3) Time as Token considers the time steps as a token where the sensor readings in certain time steps are the features. The encoder of Time Token $x^t_i$ is a one-layer vanilla Transformer encoder. (4) Sensor as Channel inputs the same data $x^t_i$ as Time as Token path. TC block represents a Temporal Dilated Convolution block with multi-scale dilated rate that extracts the locality information of time series, focusing on detailed and temporal information. (5) Irregularity gate is encoded by the first layer encoders of each path from MTSFormer in a share-weight manner, $L^1_{tc}$, $L^1_{tt}$ and $L^1_{ts}$ and pass the irregularity embedding into a gate activation function. Tanh maps the information to $[-1, 1]$ and sigmoid maps the embedding to $[0, 1]$, which allows the model adaptively preserve useful information from irregularity. (6) Multi-scale dilated convolution with different dilation rates captures the local irregular patterns by the large receptive field.
• Figure 3: MTSFormer Variants
• Figure 4: Variants Experiments on general IRTS dataset
• Figure 5: Leave-Random-Sensor-Out Experiments