ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons

TL;DR

ProbTS is presented, a benchmark tool designed as a unified platform to evaluate fundamental forecasting needs and to conduct a rigorous comparative analysis of numerous cutting-edge studies from recent years.

Abstract

Delivering precise point and distributional forecasts across a spectrum of prediction horizons represents a significant and enduring challenge in the application of time-series forecasting within various industries. Prior research on developing deep learning models for time-series forecasting has often concentrated on isolated aspects, such as long-term point forecasting or short-term probabilistic estimations. This narrow focus may result in skewed methodological choices and hinder the adaptability of these models to uncharted scenarios. While there is a rising trend in developing universal forecasting models, a thorough understanding of their advantages and drawbacks, especially regarding essential forecasting needs like point and distributional forecasts across short and long horizons, is still lacking. In this paper, we present ProbTS, a benchmark tool designed as a unified platform to evaluate these fundamental forecasting needs and to conduct a rigorous comparative analysis of numerous cutting-edge studies from recent years. We dissect the distinctive data characteristics arising from disparate forecasting requirements and elucidate how these characteristics can skew methodological preferences in typical research trajectories, which often fail to fully accommodate essential forecasting needs. Building on this, we examine the latest models for universal time-series forecasting and discover that our analyses of methodological strengths and weaknesses are also applicable to these universal models. Finally, we outline the limitations inherent in current research and underscore several avenues for future exploration.

Figures (12)

• Figure 1: An overview of ProbTS.
• Figure 2: We present a comprehensive comparison between classical models designed for long-term point forecasting and short-term distributional forecasting across various prediction horizons. It utilizes a non-Gaussianity score to highlight the complexity of the data distribution across different datasets. The aggregated performance metrics are derived from Tables \ref{['tab:short_term_fore']} and \ref{['tab:long_term_fore']}.
• Figure 3: We explore the challenges faced by current models in conducting long-term distributional forecasting, with insights drawn from Table \ref{['tab:long_term_fore']} and Table \ref{['tab:long_term_norm_crps']}. Subplot (a) shows significant error increases in AR-based models, averaged across all datasets except Traffic. Subplot (b) demonstrates how the instance-level normalization impacts performance in long-term forecasting. Subplots (c) examine how trends and seasonality impact performance across all long-term forecasting datasets and horizons.
• Figure 4: We evaluate the efficacy of time-series foundation models for various forecasting horizons and distributional estimation. Subplot (a), derived from Table \ref{['tab:ts_fm_exp_var_horizon']} and excluding results from the Electricity dataset, demonstrates the short-term forecasting capabilities and long-term error accumulation of AR-based models. Subplot (b), draw from Table \ref{['tab:fm_short_prob_fore']}, investigates short-term distributional estimation, highlighting the performance challenges of foundation models compared to CSDI in handling complex data distributions. Note that we include MOIRAI with two different context lengths, 96 and 5000, as context length significantly affects its transfer performance.
• Figure 5: We have sampled and visualized multiple time-series segments from the short-term forecasting datasets. The size of the segment window is set equal to the prediction horizon.