SeqLink: A Robust Neural-ODE Architecture for Modelling Partially Observed Time Series

TL;DR

SeqLink tackles the challenge of modeling partially observed time series by moving beyond a single ODE trajectory. It introduces a three-part architecture: an ODE auto-encoder to learn per-sample latent trajectories, a pyramidal attention mechanism to organize cross-sample relationships, and a Link-ODE to fuse multiple latent trajectories into a continuous representation for unobserved intervals. The approach yields robust representations for intermittent data and demonstrates superior forecasting and classification performance on synthetic and real-world datasets, with ablation analyses underscoring the value of cross-sample information. This framework offers a scalable path to more accurate modeling of irregular time series in domains such as healthcare and sensor networks, where long gaps and sparsity are common.

Abstract

Ordinary Differential Equations (ODE) based models have become popular as foundation models for solving many time series problems. Combining neural ODEs with traditional RNN models has provided the best representation for irregular time series. However, ODE-based models typically require the trajectory of hidden states to be defined based on either the initial observed value or the most recent observation, raising questions about their effectiveness when dealing with longer sequences and extended time intervals. In this article, we explore the behaviour of the ODE models in the context of time series data with varying degrees of sparsity. We introduce SeqLink, an innovative neural architecture designed to enhance the robustness of sequence representation. Unlike traditional approaches that solely rely on the hidden state generated from the last observed value, SeqLink leverages ODE latent representations derived from multiple data samples, enabling it to generate robust data representations regardless of sequence length or data sparsity level. The core concept behind our model is the definition of hidden states for the unobserved values based on the relationships between samples (links between sequences). Through extensive experiments on partially observed synthetic and real-world datasets, we demonstrate that SeqLink improves the modelling of intermittent time series, consistently outperforming state-of-the-art approaches.

Figures (6)

• Figure 1: (a) Bumpy irregular time series (with unobserved data highlighted using a yellow-hatched). (b) Intermittent, irregular time series (with unobserved data highlighted using a yellow-hatched). (c) Performance of ODE-RNN model on synthetic intermittent trajectory data with different lengths (100, 500, and 1000 time points) and varying levels of sparseness. The results show that ODE-RNN model is influenced by both the length of the sequence and the sparsity level (time lapse between observations).
• Figure 2: The architecture of SeqLink model with auto-encoder that generates the learned representation for each sequence, pyramidal attention module to sort the representation based on the correlations between samples, and Link-ODE to provide a continuous effective representation for the sequence based on the learned information, where $U$ (generated by ODE auto-encoder) is a set of the best-learned representations $U = \{\{u_{i}^{(1)}\}_{i=1}^{n}, \{u_{i}^{(2)}\}_{i=1}^{n}, \cdots, \{u_{i}^{(k)}\}_{i=1}^{n} \}$ for each sample $k$ at all time points $t_i$, $\alpha$: the importance weights for each latent representation. $l$: a level from $1$ to $\lvert L \rvert$ used to sort the learn representations.
• Figure 3: Architecture of Link-ODE model (the green part in Figure \ref{['fig:Two']}). Continuous-time modelling for irregular samples, where $h_0$ is the initial state of a time point $t_i$, $f(\cdot)$ is an ODE cell (update function) to solve the ODE for $h$, and $u_0^{x}$ is the previously learned trajectories from other samples $x_k$ at the same time point.
• Figure 4: Illustration of the model's behaviour in terms of the attention weights for samples from different datasets using several values of L. Each plot shows the number of samples $(n)$ assigned to the topmost layer of the pyramid after using different values of $L$. For each value, we display the average importance rates for these samples ($avg$) and the MSE (or AUC) values achieved by the model (blue line plot). Using a very high or very low value of $L$ affects the model's performance, with the optimal performance consistently achieved at a suitable value of $L$, which also depends on the size and correlation between the samples.
• Figure 5: Case Study: Sequences of two samples from PeriodicDataset_100 dataset with 10% unobserved values (a) and 40% unobserved values (b). Actual values are shown with the green dashed line, SeqLink predictions are shown with the red dashed line, and baseline (ODE-RNN) predictions are shown with the blue dashed line. SeqLink outperforms the baseline and can give more accurate predictions for the values that appear after the gap.