Variability Aware Recursive Neural Network (VARNN): A Residual-Memory Model for Capturing Temporal Deviation in Sequence Regression Modeling

TL;DR

The paper tackles non-stationary and volatile time-series regression by introducing VARNN, a residual-aware architecture that elevates recent prediction errors to a persistent memory state to recalibrate subsequent predictions. It couples a Predictor Block with a Residual-Memory Block, where the one-step innovation $e_\tau = y_\tau - \hat{y}_\tau$ updates a memory $\boldsymbol{h}_\tau$ that conditions future predictions, enabling rapid adaptation to regime shifts without heavy recurrent computation. Empirical results across appliance energy, biomedical, and air-quality datasets show VARNN delivers the lowest test MSE compared to static, dynamic, and standard recurrent baselines, with small training-test gaps and minimal computational overhead. The work demonstrates that explicit modeling of error dynamics via a memory mechanism yields robust predictions under drift and heteroscedasticity, suggesting a practical and scalable approach for time-series learning in non-stationary environments.

Abstract

Real-world time series data exhibit non-stationary behavior, regime shifts, and temporally varying noise (heteroscedastic) that degrade the robustness of standard regression models. We introduce the Variability-Aware Recursive Neural Network (VARNN), a novel residual-aware architecture for supervised time-series regression that learns an explicit error memory from recent prediction residuals and uses it to recalibrate subsequent predictions. VARNN augments a feed-forward predictor with a learned error-memory state that is updated from residuals over a short context steps as a signal of variability and drift, and then conditions the final prediction at the current time step. Across diverse dataset domains, appliance energy, healthcare, and environmental monitoring, experimental results demonstrate VARNN achieves superior performance and attains lower test MSE with minimal computational overhead over static, dynamic, and recurrent baselines. Our findings show that the VARNN model offers robust predictions under a drift and volatility environment, highlighting its potential as a promising framework for time-series learning.

Figures (6)

• Figure 1: VARNN architecture: the Residual Memory Block (green) converts the innovation $e_{\tau}$ into a residual-memory state $\mathbf{h}_{\tau}$; the Predictor Block (blue) fuses $\mathbf{x}_\tau$ with $\mathbf{h}_{\tau-1}$ and, when enabled, $\mathbf{u}_{\tau-1}$, to produce $\hat{y}_\tau$.
• Figure 2: Learning curves for VARNN-RM and VARNN-RM+AM across Appliances and Beijing datasets.
• Figure 3: Learning curves for VARNN-RM and VARNN-RM+AM across Appliances and Beijing datasets.
• Figure 4: Residual memory size vs scalar across three datasets.
• Figure 5: Train vs. validation loss on Beijing PM2.5 using VARNN-RM with ReLU vs. Tanh activations.