Are we misdiagnosing ensemble forecast reliability? On the insufficiency of Spread-Error and rank-based reliability metrics

TL;DR

This work shows that common ensemble reliability diagnostics based on spread and rank (Spread-Error, rank histogram) can misdiagnose reliability when climatological variance is biased, even in cases with high actual predictability. By analyzing a joint MVN framework, the authors derive that the expected Spread-Error difference obeys $\mathbb{E}[\delta_\tau] = - (\Delta\mu)^2 + \Delta\sigma^2 - 2\Delta\Sigma$, and that other diagnostics can fail to detect unreliability under certain covariance structures. Through idealized MVN experiments and an application to real GEPS forecasts, they demonstrate scenarios where forecasts appear reliable overall but are unreliable at specific times or events, especially for extremes. To address these limitations, they introduce the MVP (Mean-Variance-Predictability) decomposition, a simple, location-specific diagnostic using $\Delta\tilde{\mu}_{\hat{\tau}}$, $\Delta\tilde{\sigma}_{\hat{\tau}}^2$, and $\Delta\tilde{\rho}_{\hat{\tau}}$, which separates climatology from predictability and guides calibration and inflation strategies toward genuine second-order reliability.

Abstract

It has been documented that Spread-Error equality and a flat rank histogram are necessary but insufficient for demonstrating ensemble forecast reliability. Nevertheless, these metrics are heavily relied upon, both in the literature and at operational numerical weather prediction centers. In this study, we demonstrate theoretically why the Spread-Error relationship is necessary but insufficient for diagnosing reliability up to second order, even when mean bias is absent or accounted for. Assuming joint normality between ensemble members and the reference truth, we further show with idealized experiments that the same covariance structure responsible for this insufficiency also produces false diagnoses of reliability with the rank histogram and the reliability component of the continuous rank probability score. Under this structure and when the ensemble mean is meaningfully different from climatology, the truth lies among the least (most) extreme members when climatological variance is excessive (deficient) in each member. Importantly, this behavior is also shown to be plausible in operational ensemble weather forecasts. Combining these results with calibration principles from statistical postprocessing leads us to conclude that both perfect dispersion and underdispersion are ill-defined. When diagnostics are misinterpreted as indicating the latter, improper tuning can lead to further deterioration of forecast quality, even while improving Spread-Error and rank histogram behavior. To address these issues, we propose a new reliability diagnostic based on three easily computed statistics, motivated by the structure of the joint distribution of ensemble members and the reference truth up to second order. The diagnostic separates contributions to unreliability originating from climatology and predictability, enabling a more precise and robust characterization of ensemble behavior.

Figures (9)

• Figure 1: (a) Schematic of a 5-member ensemble forecast and reference true state as a function of lead time, transitioning through high, moderate, and no predictability; gray ellipses represent the climatological distributions of possible states, drawn to be the same for $X_i$ and $Y$. (b) Time series of 50-member ensemble forecasts and true states simulated with the multivariate normal distribution under each predictability regime highlighted in (a). (c) Scatter plots of the first and second ensemble member. (d) Scatter plots of the first ensemble member and the true state. The cyan histograms along the axes in (c) and (d) illustrate the climatological distribution for the corresponding ensemble member, whereas the red histograms in (d) illustrate the climatological distribution for the true state. The forecasts displayed in (b)-(d) are reliable, as they satisfy the three conditions for exchangeability up to second order described in the main text.
• Figure 2: Ensemble forecasts simulated with a multivariate normal distribution subject to a climatological mean bias and climatological variance excess: Exp1 in Table \ref{['tab:t1']}. (a) Time series of the first 75 samples of the ensemble members, ensemble mean, and true state; (b) Spread-Error difference ($\delta_\tau$; Eq. \ref{['eq:delta_tau']}), Reliability Budget ($\delta_\tau^\nu$; Eq. \ref{['eq:reliability_budget']}), and the three components of the Spread-Error decomposition in expectation (Eq. \ref{['eq:delta_tau_expect']}); (c) rank histogram; (d) reliability diagram for three exceedance events based on quantiles of the true climatological distribution. Diagnostics in (c)-(d) were computed using all $T=10^5$ samples.
• Figure 3: As in Fig. \ref{['f2']}, but for Exp2: climatological variance excess balanced by an inter-member covariance bias twice its magnitude.
• Figure 4: As in Fig. \ref{['f2']}, but for Exp3: climatological variance deficiency balanced by an inter-member covariance bias twice its magnitude.
• Figure 5: Reliability diagnostics as a function of the climatological standard deviation ratio $\sigma_x/\sigma_y$ and linear predictability bias $\Delta\rho$. Contours are constant values of $\Delta\sigma^2-2\Delta\Sigma$ (i.e., the Reliability Budget under expectation). The only coordinate corresponding to actual reliability is $\sigma_x/\sigma_y=1$ and $\Delta\rho=0$.

Theorems & Definitions (1)

• Proposition 1