e-Values for Real-Time Residential Electricity Demand Forecast Model Selection

Abstract

With the growing number of forecasting techniques and the increasing significance of forecast-based operation - particularly in the rapidly evolving energy sector - selecting the most effective forecasting model has become a critical task. Given the dynamic nature of energy forecasting, it is highly advantageous to assess the superiority of forecasting models not only retrospectively but continuously in real-time as new data and evidence becomes available, while simultaneously providing strong probabilistic guarantees for these decisions. In this work, we show that this can be achieved through the mathematical concept of e-values, which has recently gained massive attention in the field of statistics. It allows for unified construction principles for powerful tests and accurate statistical decisions, which can be evaluated at any chosen time points while maintaining an overall probabilistic error control. We extend the use of e-values by developing a simple persistence approach that dynamically combines input forecasts to generate new fused predictions. To demonstrate the performance of our method we apply it to electricity demand forecasts based on different artificial intelligence based models. Our results indicate that e-values are able to improve the accuracy and reliability of forecasts in a dynamic environment, offering a valuable tool for real-time decision-making in the energy sector.

Figures (6)

• Figure 1: Visualization of the required functions from Eq. \ref{['e-process']}. (Left) Illustration of the variance process $\widehat{V}_t$ using the data described in Section \ref{['data description']}. (Right) Sub-exponential and sub-gaussian $\psi$-functions $\psi_E$ and $\psi_N$ for different $\lambda \in [0,1)$.
• Figure 2: Histograms of the MAE of NG-RC (left), LSTM (middle) and the empirical score differences $\widehat{\delta_t}$ (right) with mean values (dotted line) in watts, and normal fit (solid line).
• Figure 3: (Left) Empirical score differences $\widehat{\delta}_t$ for the entire period. The dotted line represents the first week, which is used to calculate the scale $\sigma$. The remaining score differences are utilized for the selection procedure. (Right) Sigmoid function $f(x)$ in the range of the score differences $\widehat{\delta}_t$. The horizontal lines (dotted) represent the bounds at $-1/2$ and $1/2$. The color bar visualizes the quantity distribution density of the score differences $\widehat{\delta}_t$ along the x-axis.
• Figure 4: Comparison of different processes for two time periods: days 7 to 14 (upper plot) and day 62 to 69 (lower plot). Confidence sequence and threshold at significance level $\alpha = 0.05$. (Left) Transformed score differentials $\widetilde{\delta}_t$ (solid line) and average transformed score differentials $\widetilde{\Delta}_{t, \omega}$ (dashed dotted line), along with the corresponding confidence sequence $C_\alpha$ (shaded area). The second y-axis represents the rescaled values from the first y-axis in Watts, showing a linear relationship around zero and a compression of higher deviations. The horizontal dashed line represents the value zero. (Right)$e$-process $E_{\lambda, \omega}(t)$ (solid line) with the threshold value $2/\alpha$ (dashed line) on a logarithmic scale. The horizontal solid line represents the starting value one. Exceeding the threshold indicates that LSTM outperforms NG-RC during this period.
• Figure 5: MAEs for NG-RC (dashed line) and LSTM (dashed dotted line) over 6154 prediction points, excluded the first week of data. Shaded areas indicate model selection by the $e$-selection method for different hyperparameters $\omega$ and $\lambda$: diagonal lines for NG-RC, dotted for LSTM. Squares show periods where no clear preference is given.