Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains

Raeid Saqur, Christoph Bergmeir, Blanka Horvath, Daniel Schmidt, Frank Rudzicz, Terry Lyons

Abstract

We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of efficient classical methods. We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics, and raising questions as to whether any marginal improvements justify the significant increase in computational overhead and model complexity. We call on the community to (I) retire or substantially augment current benchmarks with datasets exhibiting a wider spectrum of non-stationarities, such as structural breaks, time-varying volatility, and concept drift, and less predictable dynamics drawn from diverse real-world domains, and (II) require every deep learning submission to include robust classical and simple baselines, appropriately chosen for the specific characteristics of the downstream tasks' time series. By doing so, we will help ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.

Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains

Abstract

We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of efficient classical methods. We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models, rendering complex deep learning architectures frequently no more performant than their classical counterparts for these specific data characteristics, and raising questions as to whether any marginal improvements justify the significant increase in computational overhead and model complexity. We call on the community to (I) retire or substantially augment current benchmarks with datasets exhibiting a wider spectrum of non-stationarities, such as structural breaks, time-varying volatility, and concept drift, and less predictable dynamics drawn from diverse real-world domains, and (II) require every deep learning submission to include robust classical and simple baselines, appropriately chosen for the specific characteristics of the downstream tasks' time series. By doing so, we will help ensure that reported gains reflect genuine scientific methodological advances rather than artifacts of benchmark selection favoring models adept at learning repetitive patterns.
Paper Structure (66 sections, 1 equation, 9 figures, 10 tables)

This paper contains 66 sections, 1 equation, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Organization:
  3. Preliminaries and Background
  4. Point Forecasting Problem.
  5. Model Categories.
  6. Evaluation Metrics.
  7. Spectrum of Time Series Characteristics.
  8. General Challenges of LLMs in Time-Series Forecasting
  9. A Thought Experiment Aided Critique of TSF LLMs
  10. The Seeking SOTA Experiment
  11. Are TSF LLMs Secretly Time Series Stein Estimators?
  12. The State of Benchmarking in AI/ML TSF Models
  13. ☛ The Danger of Cascading Errors
  14. Challenges, Weaknesses in Evaluation
  15. Improper and ad hoc error metrics.
  16. ...and 51 more sections

Figures (9)

  • Figure 1: Comparative visualization of the target variable (consistently named 'OT' in these preprocessed versions) dynamics across five key LTSF benchmark datasets. Top row: Full time series plotted with daily frequency (weekly for ILI) to highlight long-term trends and seasonalities. The vertical dashed line demarcates the train data (left) and test data (right). Bottom row: Zoomed-in views ($\pm$ largest typical forecast horizon: $\pm 30$ days for hourly ETTh2; $\pm 7.5$ days for 15-min ETTm2; $\pm 30$ days for hourly Traffic; $\pm 30$ days for daily Exchange; and $\pm 60$ weeks for weekly ILI) around the train/test split, plotted at their original data frequency. These plots reveal the nature and persistence of inherent periodic patterns across the train/test boundary.
  • Figure 2: Schematic illustration of canonical non-stationarities commonly observed in time series data.
  • Figure 3: ETTh1 Dataset: (a) Autocorrelation functions showing persistent daily periodicity (e.g., lag 24) across train/test splits for all features. (b) Full daily average view of the 'OT' target. (c) Zoomed hourly view of 'OT' around the train/test split, showing continuity of patterns. Target plots share y-max of 60.
  • Figure 4: ETTm1 Dataset: (a) Autocorrelation functions revealing strong daily periodicity (e.g., lag 96 for 15-min data) persisting across train/test splits. (b) Full daily average view of 'OT'. (c) Zoomed 15-minute view of 'OT' ($\pm 7.5$ days, i.e., $\pm 720$ steps) around the split. Target plots share y-max of 60.
  • Figure 5: ETTh2 Dataset: (a) Persistent daily periodicity evident in ACF plots across train/test splits. (b) Full daily average view of 'OT'. (c) Zoomed hourly view of 'OT' at the train/test boundary. Target plots share y-max of 60.
  • ...and 4 more figures