Online Continual Learning for Time Series: a Natural Score-driven Approach

TL;DR

The paper addresses nonstationary time series forecasting by framing online continual learning as a continuous filtering problem. It introduces NatSR, a method that unifies score-driven updates with natural gradient descent, augmented by a robust Student's t loss, a memory replay buffer, and a dynamic scale to handle regime drifts. The authors prove information-theoretic optimality of NGD in this setting, derive a bound on updates under the Student's t loss, and show that the combination with memory and scale yields strong empirical performance across multiple real datasets. NatSR offers a robust, scalable approach for online time series forecasting, enabling fast adaptation without forgetting past regimes, with potential extensions to broader online learning contexts.

Abstract

Online continual learning (OCL) methods adapt to changing environments without forgetting past knowledge. Similarly, online time series forecasting (OTSF) is a real-world problem where data evolve in time and success depends on both rapid adaptation and long-term memory. Indeed, time-varying and regime-switching forecasting models have been extensively studied, offering a strong justification for the use of OCL in these settings. Building on recent work that applies OCL to OTSF, this paper aims to strengthen the theoretical and practical connections between time series methods and OCL. First, we reframe neural network optimization as a parameter filtering problem, showing that natural gradient descent is a score-driven method and proving its information-theoretic optimality. Then, we show that using a Student's t likelihood in addition to natural gradient induces a bounded update, which improves robustness to outliers. Finally, we introduce Natural Score-driven Replay (NatSR), which combines our robust optimizer with a replay buffer and a dynamic scale heuristic that improves fast adaptation at regime drifts. Empirical results demonstrate that NatSR achieves stronger forecasting performance than more complex state-of-the-art methods.

Figures (4)

• Figure 1: Mean predictions and standard deviations of NatSR and simple Online Gradient Descent (OGD) on a noisy sinusoidal wave under two challenging conditions: (left) with outliers and (right) with changing regimes, each repeated ten times. In the outlier setting (a), OGD is destabilized and requires several iterations to recover accurate forecasts, whereas NatSR remains stable. The bottom-right panel in (a) highlights the difference in update magnitudes: OGD’s gradients grow by an order of magnitude in response to the outlier, while NatSR’s remain comparable to those from normal errors. In the regime-change setting (b), the scale rises during transitions, allowing for larger gradients and faster updates, and decreases again once the series stabilizes. This dynamic scaling, combined with second-order information from the FIM, enables NatSR to adapt rapidly to changes in both amplitude and frequency, as reflected by the smaller standard deviations during the second regime compared to OGD.
• Figure 2: Natural score of the Student's t compared to a Gaussian score. Left: score of the mean for different scales. Right: score of the scale parameter for different $\nu$.
• Figure 3: Forecasting results on three datasets: (left) ETTh1 demonstrates the model’s ability to adapt quickly; (middle) ETTm1 illustrates the stability of our model, producing less noisy predictions compared to baselines such as FSNET; (right) WTH highlights the importance of replay, as NatSR and ER achieve the best performance when revisiting previously observed input ranges.
• Figure 4: Natural Score-driven Replay (NatSR)

Theorems & Definitions (6)

• Proposition 4.1
• Proposition 4.2
• Theorem 4.1
• proof : Proof of proposition (\ref{['prop1']})
• proof : Proof of proposition (\ref{['prop2']})
• proof : Proof of Theorem \ref{['thrm']}