A Survey of Deep Learning and Foundation Models for Time Series Forecasting
John A. Miller, Mohammed Aldosari, Farah Saeed, Nasid Habib Barna, Subas Rana, I. Budak Arpinar, Ninghao Liu
TL;DR
The paper surveys deep learning and foundation-model approaches for multivariate time series forecasting, with a focus on pandemic contexts like COVID-19. It traces progress from traditional statistical methods to transformer- and graph-based architectures, and then outlines the emergence of foundation models tailored to time series, including repurposed LLMs and domain-specific pre-training, often enhanced by multimodal data and knowledge graphs. Key contributions include a taxonomy of transformer variants, a survey of graph neural networks for spatio-temporal forecasting, and a framework for incorporating knowledge and meta-evaluation metrics. The findings underscore the potential of foundation models and knowledge-enabled architectures to improve forecast accuracy and interpretability, while highlighting challenges in data diversity, short time series, and reproducibility that require standardized evaluation and robust pre-training strategies.
Abstract
Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.