Neural Chronos ODE: Unveiling Temporal Patterns and Forecasting Future and Past Trends in Time Series Data

TL;DR

The paper addresses forecasting and reconstructing time-series by learning continuous-time dynamics that exploit information in both forward and backward time directions. It introduces Neural CODE, which trains on both an IVP and a Final Value Problem to adjust ODE dynamics using past and future information, and extends this idea to recurrent architectures (CODE-RNN and CODE-BiRNN) with GRU/LSTM variants. The key contributions are the existence/uniqueness guarantees for the EVP, the development of CODE-based RNNs that outperform Neural ODE and ODE-RNN baselines, and thorough experiments on synthetic spirals and three real-world datasets showing CODE-BiRNN variants achieve the best accuracy and fastest convergence. The findings suggest that leveraging bidirectional, continuous-time updates improves missing data imputation and forward/backward extrapolation in diverse time-series domains, albeit with higher computational complexity.

Abstract

This work introduces Neural Chronos Ordinary Differential Equations (Neural CODE), a deep neural network architecture that fits a continuous-time ODE dynamics for predicting the chronology of a system both forward and backward in time. To train the model, we solve the ODE as an initial value problem and a final value problem, similar to Neural ODEs. We also explore two approaches to combining Neural CODE with Recurrent Neural Networks by replacing Neural ODE with Neural CODE (CODE-RNN), and incorporating a bidirectional RNN for full information flow in both time directions (CODE-BiRNN), and variants with other update cells namely GRU and LSTM: CODE-GRU, CODE-BiGRU, CODE-LSTM, CODE-BiLSTM. Experimental results demonstrate that Neural CODE outperforms Neural ODE in learning the dynamics of a spiral forward and backward in time, even with sparser data. We also compare the performance of CODE-RNN/-GRU/-LSTM and CODE-BiRNN/-BiGRU/-BiLSTM against ODE-RNN/-GRU/-LSTM on three real-life time series data tasks: imputation of missing data for lower and higher dimensional data, and forward and backward extrapolation with shorter and longer time horizons. Our findings show that the proposed architectures converge faster, with CODE-BiRNN/-BiGRU/-BiLSTM consistently outperforming the other architectures on all tasks.

Figures (23)

• Figure 1: The adjusted ODE dynamics, $\boldsymbol{f_\theta}$, is given by a NN with an input layer with the dimension $d+1$ to accommodate the input vector, $\boldsymbol{x} \in \mathbb{R}^d$ with $d$ features, and its time step, $t \in \mathbb{R}$.
• Figure 2: Scheme of a BiRNN unfolded through time.
• Figure 3: Scheme of a Neural CODE able to make predictions forward, by solving an IVP, and backward, by solving an FVP, in time.
• Figure 4: Training scheme of CODE-RNN and CODE-RNN unfolded through time (on the right). It uses a Neural CODE to adjust an ODE dynamics $\boldsymbol{f_\theta}$.
• Figure 5: Scheme of CODE-BiRNN and CODE-BiRNN unfolded through time.