Learning K-U-Net with constant complexity: An Application to time series forecasting

TL;DR

This work tackles the time complexity bottleneck in training deep time-series forecasting models by introducing Exponentially Weighted SGD with Momentum (EW-SGDM). EW-SGDM uses layer-wise exponential weights $W^{(l)} = S^{l-1}$ to bias gradient updates toward deeper, high-level features, yielding a constant learning-time complexity $O(1)$ for Kernel U-Net when $S \ll T$. The authors integrate EW-SGDM with Kernel U-Net to exploit the architecture's patch-kernel separation and demonstrate empirically that the approach reduces computational effort while maintaining or improving predictive accuracy on synthetic benchmarks. The results suggest practical benefits for efficient training of deep U-shaped models in time-series forecasting, with code to be released publicly.

Abstract

Training deep models for time series forecasting is a critical task with an inherent challenge of time complexity. While current methods generally ensure linear time complexity, our observations on temporal redundancy show that high-level features are learned 98.44\% slower than low-level features. To address this issue, we introduce a new exponentially weighted stochastic gradient descent algorithm designed to achieve constant time complexity in deep learning models. We prove that the theoretical complexity of this learning method is constant. Evaluation of this method on Kernel U-Net (K-U-Net) on synthetic datasets shows a significant reduction in complexity while improving the accuracy of the test set.

Figures (15)

• Figure 1: The Exponentially Weighted Gradient with Momentum (EW-SGDM) Algorithm on U-Shape Architecture. The weight $W=S^l$ is applied on the gradients of the parameters in kernels $\{\phi_1, \dots, \phi_6\}$ at each level of U-Net, where $S$ is the patch size.
• Figure 2: The phenomena of overlapping patches. The low level features are learnt $\frac{L}{S}$ more times by sliding window.
• Figure 3: Dataset 1
• Figure 4: Dataset 2
• Figure 5: Dataset 3