Ensemble-size-dependence of deep-learning post-processing methods that minimize an (un)fair score: motivating examples and a proof-of-concept solution
Christopher David Roberts
TL;DR
This work analyzes how distribution-aware ensemble post-processing that induces inter-member dependencies can undermine the fairness of the adjusted CRPS ($aCRPS$) when ensembles are finite. It first demonstrates, via a Gaussian signal-plus-noise idealization, that linear calibrations exploiting ensemble dependencies can yield apparent improvements in $aCRPS$ at the cost of reliability, highlighting the non-fairness of $aCRPS$ under dependency. The authors then compare two transformer-based post-processing approaches: the PoET ensemble transformer, which shares information across ensemble members and can improve $aCRPS$ but introduces ensemble-size sensitivity and over-dispersion; and the trajectory transformer, which applies attention across lead times per member to preserve conditional independence and achieve ensemble-size independence. Applied to ECMWF subseasonal weekly-mean $T_{2m}$ forecasts, the trajectory transformer reduces systematic biases while maintaining or improving forecast reliability across different training and real-time ensemble sizes, offering a robust proof-of-concept pathway for fair, ensemble-size–agnostic post-processing in practice.
Abstract
Fair scores reward ensemble forecast members that behave like samples from the same distribution as the verifying observations. They are therefore an attractive choice as loss functions to train data-driven ensemble forecasts or post-processing methods when large training ensembles are either unavailable or computationally prohibitive. The adjusted continuous ranked probability score (aCRPS) is fair and unbiased with respect to ensemble size, provided forecast members are exchangeable and interpretable as conditionally independent draws from an underlying predictive distribution. However, distribution-aware post-processing methods that introduce structural dependency between members can violate this assumption, rendering aCRPS unfair. We demonstrate this effect using two approaches designed to minimize the expected aCRPS of a finite ensemble: (1) a linear member-by-member calibration, which couples members through a common dependency on the sample ensemble mean, and (2) a deep-learning method, which couples members via transformer self-attention across the ensemble dimension. In both cases, the results are sensitive to ensemble size and apparent gains in aCRPS can correspond to systematic unreliability characterized by over-dispersion. We introduce trajectory transformers as a proof-of-concept that ensemble-size independence can be achieved. This approach is an adaptation of the Post-processing Ensembles with Transformers (PoET) framework and applies self-attention over lead time while preserving the conditional independence required by aCRPS. When applied to weekly mean $T_{2m}$ forecasts from the ECMWF subseasonal forecasting system, this approach successfully reduces systematic model biases whilst also improving or maintaining forecast reliability regardless of the ensemble size used in training (3 vs 9 members) or real-time forecasts (9 vs 100 members).