Accuracy Law for the Future of Deep Time Series Forecasting

TL;DR

The paper establishes an accuracy law for deep time series forecasting by linking a window-wise pattern complexity, defined in the frequency domain over prediction windows, to the minimum achievable forecasting error. Through large-scale experiments on 940 univariate series and 2,820 forecasters, the authors show an exponential relation between complexity and MSE, with a robust linear relation between Complexity and LogMSE and a zero-intercept at zero complexity. This law enables identifying saturated benchmarks and informs training strategies for large time-series models, including targeted pretraining data sampling and constructing harder out-of-distribution benchmarks. The work provides a data-driven, generalizable bound for forecasting performance, guiding future research beyond marginal benchmark gains and toward modeling complex temporal patterns.

Abstract

Deep time series forecasting has emerged as a booming direction in recent years. Despite the exponential growth of community interests, researchers are sometimes confused about the direction of their efforts due to minor improvements on standard benchmarks. In this paper, we notice that, unlike image recognition, whose well-acknowledged and realizable goal is 100% accuracy, time series forecasting inherently faces a non-zero error lower bound due to its partially observable and uncertain nature. To pinpoint the research objective and release researchers from saturated tasks, this paper focuses on a fundamental question: how to estimate the performance upper bound of deep time series forecasting? Going beyond classical series-wise predictability metrics, e.g., ADF test, we realize that the forecasting performance is highly related to window-wise properties because of the sequence-to-sequence forecasting paradigm of deep time series models. Based on rigorous statistical tests of over 2,800 newly trained deep forecasters, we discover a significant exponential relationship between the minimum forecasting error of deep models and the complexity of window-wise series patterns, which is termed the accuracy law. The proposed accuracy law successfully guides us to identify saturated tasks from widely used benchmarks and derives an effective training strategy for large time series models, offering valuable insights for future research.

Figures (9)

• Figure 1: The performance change of deep time series forecasting in the past five years on well-established benchmarks. We record the MSE averaged from four widely used forecasting horizons.
• Figure 2: Hypothesis space considered in this paper and discovered accuracy law (red star).
• Figure 3: Illustration of time and frequency distance.
• Figure 4: Experimental results from 940 series and 2,820 deep forecasters for the accuracy law. For each data point, its $x$-coordinate denotes the window-wise pattern complexity calculated by Eq. \ref{['equ:definition']} and the $y$-coordinate is the lowest forecasting error achieved among three experimental models.
• Figure 5: Comparison of our window-wise complexity with classical metrics (left) and other variations (middle), where Pearson coefficient between the predictability and performance is recorded. For the right part, the Ramsey RESET test is presented as the z-axis value; a higher bar indicates greater confidence in the linear relation; the brighter color refers to the higher Pearson coefficient.