A Measure of Predictive Sharpness for Probabilistic Models

Abstract

We introduce a sharpness functional for probabilistic models that quantifies sharpness as an intrinsic property of the probability distribution. The measure is derived based on a rank-based concentration principle that tracks upward transfers of probability mass along the rearranged profile of the predictive distribution. For finite outcome spaces, this yields a normalized sharpness measure with transparent mass--length representation and equivalent formulations as a Gini-type coefficient on the probability vector and a scaled 1-Wasserstein distance from the uniform distribution in rearranged space. We extend the functional to bounded continuous and multidimensional domains for predictive distributions with finite first moment, and establish normalization, symmetry, continuity, and monotonicity properties. The diagnostic application of the measure is illustrated with real and simulated data, and a relationship to the multivariate energy score is discussed.

Figures (21)

• Figure 1: Distributions with approximately equal variance, $\mathrm{Var}(Y) \approx 1.0$, on the 3-simplex (n=4). Each point represents a discrete distribution over four outcomes with $\mathrm{Var}(Y) \approx 1.0$. In this setting, a distribution such as $\{0.19,\ 0.32,\ 0.3,\ 0.19\}$ obtains a low of $\mathrm{S}(P) \approx 0.17$, while a high of $\mathrm{S}(P) \approx 0.89$ is given by $\{0.86,\ 0,\ 0.02,\ 0.12\}$.
• Figure 2: Plots for select pdfs in Table \ref{['tbl-two']} and the integral components of $S(f)$.
• Figure 3: Concentration plots for three-dimensional probability density functions defined on $\Omega = [0,2]^3$. For illustration, the first two distributions (on the left) are defined based on different types of concentration patterns occurring in four equal areas of the domain (full exclusion, slower and faster concentration, and uniformity), showing how these changes impact the shape of the plot. The x-axis measures contribution to the sharpness score, and the 0.5 intercept marks when the contribution falls to less than half of the total possible (obtained by full exclusion). The plots display fractions of the domain fully excluded, piecewise constant areas, the rate of concentration growth, and the amount of probability mass packed into fractions of the domain.
• Figure 4: Lorenz curve representation of the three 3D pdfs displayed in Figure \ref{['fig:3D']}.
• Figure 5: Global 24-hour precipitation forecasts from IFS ensemble dataset, initialized 1--8 January 2021. Sharpness is shown alongside CRPS, 90% prediction interval width, and ensemble standard deviation.

Theorems & Definitions (21)

• Definition 2.1
• Remark
• Proposition 2.1
• proof
• Proposition 2.2
• Proposition 2.3
• Proposition 3.1
• proof
• proof
• Proposition S2.1