Analyzing the retraining frequency of global forecasting models: towards more stable forecasting systems

TL;DR

This study investigates how retraining frequency affects the stability of global forecasting models across large-scale datasets (M4, M5, VN1). It introduces Scaled Multi-Quantile Change (SMQC), a model-agnostic metric for probabilistic stability, and demonstrates that less frequent retraining often preserves or enhances both forecast stability and accuracy, especially for global models. The work contrasts global models with local baselines, showing that cross-learning yields robustness to update frequency and can reduce computational costs substantially. Practical guidelines for stability-aware retraining schedules are provided, with daily data benefiting from roughly 21–30 day cycles and weekly data up to 8 weeks, along with a shift toward prioritizing stability alongside accuracy. The findings have implications for deploying sustainable, stable forecasting systems in retail, energy, and supply-chain contexts.

Abstract

Forecast stability, that is, the consistency of predictions over time, is essential in business settings where sudden shifts in forecasts can disrupt planning and erode trust in predictive systems. Despite its importance, stability is often overlooked in favor of accuracy. In this study, we evaluate the stability of point and probabilistic forecasts across several retraining scenarios using three large forecastingdatasets and ten different global forecasting models. To analyze stability in the probabilistic setting, we propose a new model-agnostic, distribution-free, and scale-free metric that measuresprobabilistic stability: the Scaled Multi-Quantile Change (SMQC). The results show that less frequent retraining not only preserves but often improves forecast stability, challenging the need for frequent retraining. Moreover, the study shows that accuracy and stability are not necessarily conflicting objectives when adopting a global modeling approach. The study promotes a shift toward stability-aware forecasting practices, proposing a new metric to evaluate forecast stability effectively in probabilistic settings, and offering practical guidelines for building more stable and sustainable forecasting systems.

Figures (21)

• Figure 1: RMSSE and sMAPC results for each global method and retrain scenario combination in relative terms with respect to the baseline scenario, $r = 7$ for the M4 and M5 datasets, and $r = 1$ for the VN1 dataset.
• Figure 2: SMQL and SMQC results for each method and retrain scenario combination in relative terms with respect to the baseline scenario, $r = 7$ for the M4 and M5 datasets, and $r = 1$ for the VN1 dataset.
• Figure 3: RMSSE, sMAPC, SMQL, and SMQC results for ETS method on the M5 dataset for each retrain scenario combination in relative terms with respect to the baseline scenario, $r = 7$.
• Figure 4: The optimal retraining frequency identified for each forecasting objective, and for each series in the datasets considering all the models.
• Figure 5: RMSSE and sMAPC results for each local method and retrain scenario combination in relative terms with respect to the baseline scenario, r = 7 for the M4 and M5 datasets, and r = 1 for the VN1 dataset.