TimeSynth: A Framework for Uncovering Systematic Biases in Time Series Forecasting

TL;DR

TimeSynth addresses the question of when simple linear models versus nonlinear architectures truly excel in time-series forecasting by introducing a principled synthetic framework that mimics real-world dynamics. It defines three signal families—Drift Harmonic, SPM-Harmonic, and DPM-Harmonic—derived from real data and paired with rigorous evaluation paradigms that include noise and distribution shifts. The study demonstrates a consistent bias in linear models, which collapse to simple oscillations or the global mean, while nonlinear models, especially CNNs and Transformers, better preserve amplitude, frequency, and phase, with robustness varying by signal family under noise and shifts. The work provides a principled baseline for evaluating forecasting approaches, emphasizing frequency-domain and phase-alignment metrics to complement traditional amplitude errors and guiding principled model selection and benchmarking in applied forecasting tasks.

Abstract

Time series forecasting is a fundamental tool with wide ranging applications, yet recent debates question whether complex nonlinear architectures truly outperform simple linear models. Prior claims of dominance of the linear model often stem from benchmarks that lack diverse temporal dynamics and employ biased evaluation protocols. We revisit this debate through TimeSynth, a structured framework that emulates key properties of real world time series,including non-stationarity, periodicity, trends, and phase modulation by creating synthesized signals whose parameters are derived from real-world time series. Evaluating four model families Linear, Multi Layer Perceptrons (MLP), Convolutional Neural Networks (CNNs), and Transformers, we find a systematic bias in linear models: they collapse to simple oscillation regardless of signal complexity. Nonlinear models avoid this collapse and gain clear advantages as signal complexity increases. Notably, Transformers and CNN based models exhibit slightly greater adaptability to complex modulated signals compared to MLPs. Beyond clean forecasting, the framework highlights robustness differences under distribution and noise shifts and removes biases of prior benchmarks by using independent instances for train, test, and validation for each signal family. Collectively, TimeSynth provides a principled foundation for understanding when different forecasting approaches succeed or fail, moving beyond oversimplified claims of model equivalence.

Figures (32)

• Figure 1: Visualization of 5-second segments from each synthesized signal family.
• Figure 2: Per-series amplitude error (MAE) on clean Drift Harmonic, SPM-Harmonic, and DPM-Harmonic signals.
• Figure 3: Horizon-wise MSE evolution across prediction steps for linear and nonlinear Models.
• Figure 4: Error distributions on clean signals: (a) frequency error, (b) phase error.
• Figure 5: Phase error under noise conditions: (a) high-noise vulnerability across families, (b) Effect of noise on phase of SPM Harmonic signals.