Beyond Accuracy: A Stability-Aware Metric for Multi-Horizon Forecasting
Chutian Ma, Grigorii Pomazkin, Giacinto Paolo Saggese, Paul Smith
TL;DR
The paper addresses the limitation of standard forecasting approaches that optimize accuracy alone, by introducing the forecast accuracy and coherence (AC) score that jointly quantifies multi-horizon forecast accuracy and stability. The AC score combines an energy-score-based accuracy term with a stability term across forecast origins, parameterized by a stability weight $\\lambda$ and horizon-aware weights $\\omega(h)$, yielding a flexible, decision-aware objective $S_{AC}$. The authors implement the AC score as a differentiable objective to train a SARIMA model and demonstrate substantial improvements in forecast stability (roughly a 75% reduction in volatility) with comparable or improved multi-horizon accuracy on a subset of the M4 Hourly dataset. This framework supports customizable, architecture-agnostic evaluation and training for probabilistic multi-horizon forecasts and points to extensions to more complex models and refined penalties for justified revisions.
Abstract
Traditional time series forecasting methods optimize for accuracy alone. This objective neglects temporal consistency, in other words, how consistently a model predicts the same future event as the forecast origin changes. We introduce the forecast accuracy and coherence score (forecast AC score for short) for measuring the quality of probabilistic multi-horizon forecasts in a way that accounts for both multi-horizon accuracy and stability. Our score additionally provides for user-specified weights to balance accuracy and consistency requirements. As an example application, we implement the score as a differentiable objective function for training seasonal ARIMA models and evaluate it on the M4 Hourly benchmark dataset. Results demonstrate substantial improvements over traditional maximum likelihood estimation. Our AC-optimized models achieve a 75\% reduction in forecast volatility for the same target timestamps while maintaining comparable or improved point forecast accuracy.