Time Series Foundation Models: Benchmarking Challenges and Requirements

TL;DR

The paper investigates how Time Series Foundation Models (TSFMs) are evaluated and identifies critical pitfalls of current benchmarking, including test-set contamination from multi-purpose data usage and memorization of global patterns induced by external shocks. It analyzes 15 TSFMs to trace data lineage and demonstrates how leakage can inflate performance, especially for larger models, calling for evaluation on truly out-of-sample future data. The authors argue for a principled benchmark design that enforces unambiguous data splits, time- and domain-cross-validation, and flexible pre-training data, potentially via a continuously advancing global temporal split. They propose concrete benchmark requirements and discuss options like held-out datasets, synthetic data, and prospective real-world evaluation platforms to safeguard integrity and enable fair, scalable comparison. The work provides a roadmap to avoid an LLM-style evaluation crisis in TSFMs and to improve reliability of zero-shot forecasting research and practice.

Abstract

Time Series Foundation Models (TSFMs) represent a new paradigm for time series forecasting, offering zero-shot forecasting capabilities without the need for domain-specific pre-training or fine-tuning. However, as with Large Language Models (LLMs), evaluating TSFMs is tricky, as with ever more extensive training sets, it becomes more and more challenging to ensure the integrity of benchmarking data. Our investigation of existing TSFM evaluation highlights multiple challenges, ranging from the representativeness of the benchmark datasets, over the lack of spatiotemporal evaluation, to risks of information leakage due to overlapping and obscure datasets, and the memorization of global patterns caused by external shocks like economic crises or pandemics. Our findings reveal widespread confusion regarding data partitions, risking inflated performance estimates and incorrect transfer of global knowledge to local time series. We argue for the development of robust evaluation methodologies to prevent pitfalls already observed in LLM and classical time series benchmarking, and call upon the research community to design new, principled approaches, such as evaluations on truly out-of-sample future data, to safeguard the integrity of TSFM assessment.

Figures (8)

• Figure 1: Lineage of dataset collections (left) used for training and evaluating recent Time Series Foundation Models (right). Typically, a collection contains multiple different datasets. Lines indicate cases where at least one dataset of a collection was used for pre-training, train/test, or zero-shot evaluation.
• Figure 2: Global pattern memorization of TSFM A when trained on time series 1 and zero-shot forecasting time series 2.
• Figure 3: Evaluation strategies along the time or the domain dimension.
• Figure 4: Evaluation along time and domain dimension.
• Figure 5: Most common frequencies in investigated datasets.