SynTSBench: Rethinking Temporal Pattern Learning in Deep Learning Models for Time Series

TL;DR

SynTSBench offers a principled, synthetic-data framework for evaluating time series forecasting by isolating core temporal features, robustness, and theoretical optima. By formalizing a decomposition $y(t)=x(t)+n(t)$ and constructing controllable patterns with known optimal forecasts, the framework enables direct comparison against mathematical baselines and across a wide range of architectures, including zero-shot foundation models. The empirical study reveals that no single architecture excels across all pattern types, and robustness to noise/anomalies varies markedly by design (e.g., channel-dependent models vs. transformer-based ones), with strong real-world generalization validated on eight datasets. Overall, SynTSBench provides both diagnostic insight and practical guidance for model selection and architectural innovation in industrial forecasting contexts.

Abstract

Recent advances in deep learning have driven rapid progress in time series forecasting, yet many state-of-the-art models continue to struggle with robust performance in real-world applications, even when they achieve strong results on standard benchmark datasets. This persistent gap can be attributed to the black-box nature of deep learning architectures and the inherent limitations of current evaluation frameworks, which frequently lack the capacity to provide clear, quantitative insights into the specific strengths and weaknesses of different models, thereby complicating the selection of appropriate models for particular forecasting scenarios. To address these issues, we propose a synthetic data-driven evaluation paradigm, SynTSBench, that systematically assesses fundamental modeling capabilities of time series forecasting models through programmable feature configuration. Our framework isolates confounding factors and establishes an interpretable evaluation system with three core analytical dimensions: (1) temporal feature decomposition and capability mapping, which enables systematic evaluation of model capacities to learn specific pattern types; (2) robustness analysis under data irregularities, which quantifies noise tolerance thresholds and anomaly recovery capabilities; and (3) theoretical optimum benchmarking, which establishes performance boundaries for each pattern type-enabling direct comparison between model predictions and mathematical optima. Our experiments show that current deep learning models do not universally approach optimal baselines across all types of temporal features.The code is available at https://github.com/TanQitai/SynTSBench

Figures (36)

• Figure 1: Overview of SynTSBench, a synthetic data-based evaluation framework for time series forecasting models. The framework generates controlled time series data from basic univariate components to complex multivariate patterns with defined relationships. This approach enables systematic assessment of model capabilities across temporal pattern learning, robustness, and dependency modeling, complex pattern recognition, and so on.
• Figure 2: Visualization of diverse trend functions used in our benchmark to systematically evaluate model performance across a wide range of temporal patterns.
• Figure 3: Relative prediction difficulty by trend type. Higher values indicate easier prediction (log scale)
• Figure 4: Comparison of MSE$_{Obs}$ across noise levels for twelve forecasting models.
• Figure 5: Comparison of MSE$_{True}$ across noise levels for twelve forecasting models.