IRNN: Innovation-driven Recurrent Neural Network for Time-Series Data Modeling and Prediction

TL;DR

This work tackles time-series forecasting with nonlinear dynamics by introducing Innovation-driven RNNs (IRNNs) that integrate past prediction innovations into hidden-state updates. By drawing from Kalman Filter concepts, it adds $e_{t-1} = y_{t-1} - \hat{y}_{t-1}$ as an input, and proposes a tailored training scheme IU-BPTT to handle the coupling between innovations and network parameters. The approach extends to innovation-driven GRU and LSTM variants (IGRU, ILSTM), and demonstrates substantial multistep prediction gains on ETT benchmarks with only a minor increase in parameters and training cost. These results highlight the practical potential of KF-inspired feedback in neural time-series models for improved accuracy in real-world forecasting tasks.

Abstract

Many real-world datasets are time series that are sequentially collected and contain rich temporal information. Thus, a common interest in practice is to capture dynamics of time series and predict their future evolutions. To this end, the recurrent neural network (RNN) has been a prevalent and effective machine learning option, which admits a nonlinear state-space model representation. Motivated by the resemblance between RNN and Kalman filter (KF) for linear state-space models, we propose in this paper Innovation-driven RNN (IRNN), a novel RNN architecture tailored to time-series data modeling and prediction tasks. By adapting the concept of "innovation" from KF to RNN, past prediction errors are adopted as additional input signals to update hidden states of RNN and boost prediction performance. Since innovation data depend on network parameters, existing training algorithms for RNN do not apply to IRNN straightforwardly. Thus, a tailored training algorithm dubbed input updating-based back-propagation through time (IU-BPTT) is further proposed, which alternates between updating innovations and optimizing network parameters via gradient descent. Experiments on real-world benchmark datasets show that the integration of innovations into various forms of RNN leads to remarkably improved prediction accuracy of IRNN without increasing the training cost substantially.

Figures (8)

• Figure 1: Illustration of single-layer RNN (\ref{['fig:RNN_structure']}) and single-layer IRNN (\ref{['fig:IRNN_structure']}) architecture for time-series data. The prediction $\hat{\bm{y}}_t$ is made at each time instance with continuous input $\bm{u}_t$ to generate $\bm{e}_t$. The calculation details of $\bm{x}_t$ are displayed within the yellow rectangle.
• Figure 2: IU-BPTT for training single-layer IRNN. The updating of network parameters (left) and the updating of innovations (right) alternate in a cyclic manner. Dashed and red solid arrows indicate inactive and active data flow, respectively.
• Figure 4: Visualization of prediction results. We display the prediction results of LSTM and ILSTM on continuous data with length of 80 in two different time scales. (a) Prediction on ETTh2. (b) Prediction on ETTm2.
• Figure 5: Training and validation loss curves on ETTh1. MSE losses are presented on a logarithmic scale. The minimal MSE in each loss curve is marked with a horizontal dashed line. (a) Single-layer RNN and IRNN. (b) GRU and IGRU. (c) LSTM and ILSTM.
• Figure 6: Box plot of total training time for Single-layer IRNN, IGRU, ILSTM, and their RNN counterparts on ETTh1.