Scoring rule nets: beyond mean target prediction in multivariate regression

TL;DR

This paper tackles multivariate probabilistic regression by introducing Conditional CRPS (CCRPS), a multivariate extension of CRPS that is more sensitive to correlation than the Energy Score. It develops closed-form CCRPS expressions for common distributions and constructs CCRPS-based ANN losses for multivariate Gaussian mixtures, along with an Energy Score ensemble loss for differentiable training. Through synthetic and real-world experiments, CCRPS-based methods often outperform maximum likelihood estimation and achieve performance on par with nonparametric methods like Distributional Random Forest. The work demonstrates that CCRPS can improve sharpness while maintaining calibration, providing a practical and principled route for reliable multivariate probabilistic forecasting.

Abstract

Probabilistic regression models trained with maximum likelihood estimation (MLE), can sometimes overestimate variance to an unacceptable degree. This is mostly problematic in the multivariate domain. While univariate models often optimize the popular Continuous Ranked Probability Score (CRPS), in the multivariate domain, no such alternative to MLE has yet been widely accepted. The Energy Score - the most investigated alternative - notoriously lacks closed-form expressions and sensitivity to the correlation between target variables. In this paper, we propose Conditional CRPS: a multivariate strictly proper scoring rule that extends CRPS. We show that closed-form expressions exist for popular distributions and illustrate their sensitivity to correlation. We then show in a variety of experiments on both synthetic and real data, that Conditional CRPS often outperforms MLE, and produces results comparable to state-of-the-art non-parametric models, such as Distributional Random Forest (DRF).

Figures (4)

• Figure 1: Visualization of Conditional CRPS, using $d = 2$ and $\mathcal{T} = \{(2, \{1\}), (1, \{2\})\}$. CCRPS evaluates an observed multivariate distribution sample by computing the distribution's univariate conditionals, conditioned on observations for other variates.
• Figure 2: Plot of mean score values against the deviation of a predicted distribution parameter from the true distribution parameter. We evaluate three strictly proper scoring rules with respect to the deviation of the predicted mean, standard deviation or correlation coefficient from the data distribution ($\mu_\text{true} = 1$, $\sigma_\text{true} = 1$ and $\rho_\text{true} = 0.4$). See Appendix D.
• Figure 3: The output layer of a CCRPS network defines a set of weights, mean vectors and positive definite matrices, via a combination of activation functions and Cholesky parameterizations. These parameters define the predicted multivariate mixture Gaussian distribution, which is evaluated against an observation via CCRPS loss.
• Figure 4: NO$_2$ (in ng/m$_3$) and PM$_{2.5}$ (in p.p.b.) predictions of the best four models for an entry in the "air" experiment testing set. The red dot denotes the target measurement.

Theorems & Definitions (16)

• definition thmcounterdefinition: Conditional CRPS
• theorem thmcountertheorem: Propriety of Conditional CRPS
• theorem thmcountertheorem: Strict propriety of Conditional CRPS
• theorem thmcountertheorem: CCRPS expression for multivariate mixture Gaussians
• lemma thmcounterlemma: Finiteness of Conditional CRPS for correct predictions
• proof
• lemma thmcounterlemma: Propriety of Conditional CRPS
• proof
• lemma thmcounterlemma: Probability of difference in conditionals and strict propriety
• proof