Aggregating distribution forecasts from deep ensembles

TL;DR

This work systematically compares probability- and quantile-based aggregation methods for three neural network-based approaches with different forecast distribution types as output and proposes a general quantile aggregation framework for deep ensembles that allows for corrections of systematic deficiencies and performs well in a variety of settings.

Abstract

The importance of accurately quantifying forecast uncertainty has motivated much recent research on probabilistic forecasting. In particular, a variety of deep learning approaches has been proposed, with forecast distributions obtained as output of neural networks. These neural network-based methods are often used in the form of an ensemble, e.g., based on multiple model runs from different random initializations or more sophisticated ensembling strategies such as dropout, resulting in a collection of forecast distributions that need to be aggregated into a final probabilistic prediction. With the aim of consolidating findings from the machine learning literature on ensemble methods and the statistical literature on forecast combination, we address the question of how to aggregate distribution forecasts based on such `deep ensembles'. Using theoretical arguments and a comprehensive analysis on twelve benchmark data sets, we systematically compare probability- and quantile-based aggregation methods for three neural network-based approaches with different forecast distribution types as output. Our results show that combining forecast distributions from deep ensembles can substantially improve the predictive performance. We propose a general quantile aggregation framework for deep ensembles that allows for corrections of systematic deficiencies and performs well in a variety of settings, often superior compared to a linear combination of the forecast densities. Finally, we investigate the effects of the ensemble size and derive recommendations of aggregating distribution forecasts from deep ensembles in practice.

Figures (19)

• Figure 1: PDF, CDF and quantile function of two normally distributed forecasts $F_1$ and $F_2$ ($\mu_1 = 7$, $\mu_2 = 10$, $\sigma_1 = \sigma_2 = 1$) together with forecasts aggregated via the methods presented in Section \ref{['sec:aggregation']}. V$_a^{=}$ and V$_a^{w}$ use the intercept $a = -6$, V$_0^{w}$ and V$_a^{w}$ the weight $w_0 = 0.65$.
• Figure 2: Graphical illustration of the general framework for NN-based probabilistic forecasting.
• Figure 3: PDF, CDF and quantile function of two HEN forecasts $F_1$ and $F_2$ together with forecasts aggregated via the LP and V$_0^{=}$. The dashed vertical lines indicate the binning with respect to $F_1$, $F_2$ and $F_w$ for the CDF plot and with respect to $Q_w$ in the quantile function plot.
• Figure 4: PIT histograms of Bayesian DEs and the aggregation methods for the three NN variants and the Kin8nm data. The ensembles are of size 10.
• Figure 5: Evaluation metrics of Bayesian DEs and the aggregation methods for the three NN variants and the Kin8nm data. Note the different scales on the vertical axis.