Postprocessing of Ensemble Weather Forecasts Using Permutation-invariant Neural Networks

TL;DR

This work demonstrates that permutation-invariant neural networks can be effective for postprocessing ensemble weather forecasts by treating ensembles as unordered sets rather than relying solely on summary statistics. It compares set-pooling and set-transformer architectures against traditional baselines (EMOS, DRN, BQN) across wind gust and surface temperature tasks, finding state-of-the-art performance in several cases while sometimes showing similar gains to existing methods. A permutation-based importance analysis reveals that much of the predictive information resides in a few ensemble-internal degrees of freedom, particularly the ensemble mean, with variability in the importance of other features across predictors and lead times. The study highlights both the promise and the computational costs of ensemble-valued postprocessing and suggests using these models to guide data reduction and future ensemble design. Overall, permutation-invariant approaches provide a flexible framework to exploit ensemble information for probabilistic weather forecasting with demonstrated gains in calibration and sharpness across multiple benchmarks.

Abstract

Statistical postprocessing is used to translate ensembles of raw numerical weather forecasts into reliable probabilistic forecast distributions. In this study, we examine the use of permutation-invariant neural networks for this task. In contrast to previous approaches, which often operate on ensemble summary statistics and dismiss details of the ensemble distribution, we propose networks that treat forecast ensembles as a set of unordered member forecasts and learn link functions that are by design invariant to permutations of the member ordering. We evaluate the quality of the obtained forecast distributions in terms of calibration and sharpness and compare the models against classical and neural network-based benchmark methods. In case studies addressing the postprocessing of surface temperature and wind gust forecasts, we demonstrate state-of-the-art prediction quality. To deepen the understanding of the learned inference process, we further propose a permutation-based importance analysis for ensemble-valued predictors, which highlights specific aspects of the ensemble forecast that are considered important by the trained postprocessing models. Our results suggest that most of the relevant information is contained in a few ensemble-internal degrees of freedom, which may impact the design of future ensemble forecasting and postprocessing systems.

Figures (5)

• Figure 1: Set pooling architecture (a), consisting of encoder and decoder MLPs, and set transformer (b), featuring attention blocks and intermediate MLPs with residual connections. While the encoder-decoder architecture admits interactions between members only inside the pooling step, the set transformer admits information transfer between the members in each attention step.
• Figure 2: PIT histograms of the postprocessing models for EUPPBench 11-member reforecast and 51-member forecast ensembles (left) and 20-member wind gust forecasts (right),
• Figure 3: Permutation feature importance for summary-based networks (top) and permutation-invariant models (bottom) for EUPPBench and wind gust postprocessing. Bar heights indicate the median of an ensemble of 20 separate models, the error bars depict the IQR. Predictors named ens in the top figure correspond to the primary predictors t2m and VMAX-10M, respectively. The suffix sd indicates the ensemble standard deviation of the predictor.
• Figure 4: Importance of ensemble-internal DOFs for wind gust postprocessing. Bar charts show importance ratios $\chi(\tilde{\Pi}_{\{\pi_b\}|s}^{(i)}, \tilde{\Pi}^{(i)}_\pi)$ for selected summary statistics $s$, and heatmaps display the Spearman rank correlation between the summary statistics computed on the original dataset and the same statistics after conditional shuffling with respect to the different summary statistics. Bar heights indicate the median of an ensemble of 20 separate models, the error bars depict the IQR.
• Figure 5: Importance of ensemble-internal DOFs for temperature postprocessing. Same as Fig. \ref{['fig:importance:ensdofs:gusts']}.