FreIE: Low-Frequency Spectral Bias in Neural Networks for Time-Series Tasks

TL;DR

This paper addresses the pervasive spectral bias observed when neural networks learn time-series data, showing that low-frequency components are typically captured first across architectures. It introduces FreLE, a plug-and-play loss unit comprising explicit frequency regularization and implicit frequency regularization to balance and denoise frequency information during training, backed by Fourier-domain analysis and denoising strategies. Empirical evaluations across seven real-world datasets demonstrate FreLE's superior performance, with extensive ablations confirming the necessity of both components and robustness to hyperparameters. The proposed method has practical implications for improving long-term time-series forecasting by leveraging frequency-domain priors, and code is publicly available for reproducibility.

Abstract

The inherent autocorrelation of time series data presents an ongoing challenge to multivariate time series prediction. Recently, a widely adopted approach has been the incorporation of frequency domain information to assist in long-term prediction tasks. Many researchers have independently observed the spectral bias phenomenon in neural networks, where models tend to fit low-frequency signals before high-frequency ones. However, these observations have often been attributed to the specific architectures designed by the researchers, rather than recognizing the phenomenon as a universal characteristic across models. To unify the understanding of the spectral bias phenomenon in long-term time series prediction, we conducted extensive empirical experiments to measure spectral bias in existing mainstream models. Our findings reveal that virtually all models exhibit this phenomenon. To mitigate the impact of spectral bias, we propose the FreLE (Frequency Loss Enhancement) algorithm, which enhances model generalization through both explicit and implicit frequency regularization. This is a plug-and-play model loss function unit. A large number of experiments have proven the superior performance of FreLE. Code is available at https://github.com/Chenxing-Xuan/FreLE.

Figures (6)

• Figure 1: The spectral loss graph for a 2-DNN across different synthetic datasets. The line graph represents represents the frequency comparison between the original data and the output data in the final iteration, and the heatmap illustrates the decrease in the RMSE loss metric as the iterations progress, showing how the three primary frequencies change with the number of iterations
• Figure 2: The spectral loss graph for LSTM across different $\sigma$.
• Figure 3: Under different synthetic datasets, frequency amplitude loss diagram of $\sigma_{\mathrm{ricker}}$ with $a=1$.
• Figure 4: The framework diagram of FreLE, where IFR Unit refers to the Implicit Frequency Regularization module.
• Figure 5: Hyperparameter Sensitivity.