Asymmetric Penalties Underlie Proper Loss Functions in Probabilistic Forecasting

TL;DR

This work reveals that many common proper loss functions used in probabilistic forecasting penalize errors asymmetrically across scale and location, with sharp, quantifiable distinctions among scale- and exponential-family settings. By analyzing scale, location, and exponential families, the authors derive general rules: CRP and energy losses tend to favor underestimating scale, quadratic and DS favor overestimating scale, while logarithmic loss behaves disparately depending on the family; location-family results show symmetry for several losses under translation, with notable caveats for log loss in asymmetric or non-symmetric bases, and a detailed treatment of exponential families clarifies when logarithmic loss favors over- vs underestimation. Empirically, these asymmetries manifest in Covid-19 mortality, retail sales, temperature extremes, and synthetic data, influencing forecast ranking and hedging behavior under distribution shift. The hedging analysis demonstrates that when divergences are symmetric and rescalable, one can choose a hedge scale to reduce expected loss under test-time shifts, with explicit guidance for CRP, quadratic, and logarithmic losses in various exponential-family settings. Overall, the paper provides a unifying theory of asymmetry in proper losses, with practical implications for forecast evaluation, forecast design, and robustness to distribution shift.

Abstract

Accurately forecasting the probability distribution of phenomena of interest is a classic and ever more widespread goal in statistics and decision theory. In comparison to point forecasts, probabilistic forecasts aim to provide a more complete and informative characterization of the target variable. This endeavor is only fruitful, however, if a forecast is "close" to the distribution it attempts to predict. The role of a loss function -- also known as a scoring rule -- is to make this precise by providing a quantitative measure of proximity between a forecast distribution and target random variable. Numerous loss functions have been proposed in the literature, with a strong focus on proper losses, that is, losses whose expectations are minimized when the forecast distribution is the same as the target. In this paper, we show that a broad class of proper loss functions penalize asymmetrically, in the sense that underestimating a given parameter of the target distribution can incur larger loss than overestimating it, or vice versa. Our theory covers many popular losses, such as the logarithmic, continuous ranked probability, quadratic, and spherical losses, as well as the energy and threshold-weighted generalizations of continuous ranked probability loss. To complement our theory, we present experiments with real epidemiological, meteorological, and retail forecast data sets. Further, as an implication of the loss asymmetries revealed by our work, we show that hedging is possible under a setting of distribution shift.

Figures (4)

• Figure 1: Expected logarithmic and CRP losses for a fixed standard normal target and normal forecasts with varying location $\mu$ and scale $\sigma$. A lighter color represents a lower loss, with minimum achieved at the star, where the forecast distribution is also a standard normal. When the location is correctly specified, logarithmic loss penalizes underestimating the scale more than overestimating it, whereas the opposite is true for CRP. When the scale is correctly specified, both losses penalize symmetrically on the location.
• Figure 5: Average divergence for 30 models in CMIP6, in cases where they are initially overdispersed (left panel) or underdispersed (right panel). For each forecast distribution $F_{a\theta}$, and target distribution $G_\theta$, we compare divergences $d(F_{a\theta}, G_\theta)$ and $d(F_{\theta/a}, G_\theta)$ to see whether the given divergence prefers overdispersion to underdispersion. The average Cramér divergence is improved for all 30 models when initially overdispersed forecasts are made underdispersed, and the average KL divergence is improved for all 30 models when underdispersed forecasts are made overdispersed.
• Figure 6: Expected losses for a zero-mean unit-variance asymmetric Laplace target, and forecasts of the same family with varying location $\mu$ and scale $\sigma$. Distributions are right skewed (top panel; $p=0.2$), symmetric (middle; $p=1/2$), or left skewed (bottom; $p=0.8$). A lighter color represents a lower loss, with minimum achieved at the star.
• Figure 7: The function $\ell(p_{\eta_1}, p_\eta) - \ell(p_{\eta_2}, p_\eta)$ for normal densities with distinct fixed values of $\sigma_2$ (solid), the roots given by Theorem \ref{['thm:exponential-root']} in terms of the Lambert function (dashed) and the multiplicative inverse of the largest of the two roots (dotted).

Theorems & Definitions (26)

• Theorem 1: Scale family
• Theorem 2: Exponential family
• Theorem 3: Location family
• Lemma 1
• Lemma 2
• proof
• proof : Proof of Theorem \ref{['thm:scale-family']}
• Lemma 3
• proof
• proof : Proof of Theorem \ref{['thm:location-family']}