Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrated Multivariate Distributional Regression with Pre-Rank Regularization

Aya Laajil, Elnura Zhalieva, Naomi Desobry, Souhaib Ben Taieb

TL;DR

The paper tackles the challenge of achieving calibrated joint predictive distributions in multivariate regression. It introduces a differentiable pre-rank regularizer that enforces multivariate calibration during training by penalizing non-uniform projected PIT values with respect to chosen pre-ranks, including a novel PCA-based pre-rank. Through simulations and 18 real-world datasets, the authors show that pre-rank regularization substantially improves multivariate calibration (lower PCE) while preserving predictive accuracy (NLL and ES), and that the PCA pre-rank uncovers dependence-structure misspecifications not detected by existing pre-ranks. This framework enables calibration in training time, provides flexible diagnostics via multiple pre-ranks, and offers practical gains for reliable multivariate probabilistic forecasting.

Abstract

The goal of probabilistic prediction is to issue predictive distributions that are as informative as possible, subject to being calibrated. Despite substantial progress in the univariate setting, achieving multivariate calibration remains challenging. Recent work has introduced pre-rank functions, scalar projections of multivariate forecasts and observations, as flexible diagnostics for assessing specific aspects of multivariate calibration, but their use has largely been limited to post-hoc evaluation. We propose a regularization-based calibration method that enforces multivariate calibration during training of multivariate distributional regression models using pre-rank functions. We further introduce a novel PCA-based pre-rank that projects predictions onto principal directions of the predictive distribution. Through simulation studies and experiments on 18 real-world multi-output regression datasets, we show that the proposed approach substantially improves multivariate pre-rank calibration without compromising predictive accuracy, and that the PCA pre-rank reveals dependence-structure misspecifications that are not detected by existing pre-ranks.

Calibrated Multivariate Distributional Regression with Pre-Rank Regularization

TL;DR

The paper tackles the challenge of achieving calibrated joint predictive distributions in multivariate regression. It introduces a differentiable pre-rank regularizer that enforces multivariate calibration during training by penalizing non-uniform projected PIT values with respect to chosen pre-ranks, including a novel PCA-based pre-rank. Through simulations and 18 real-world datasets, the authors show that pre-rank regularization substantially improves multivariate calibration (lower PCE) while preserving predictive accuracy (NLL and ES), and that the PCA pre-rank uncovers dependence-structure misspecifications not detected by existing pre-ranks. This framework enables calibration in training time, provides flexible diagnostics via multiple pre-ranks, and offers practical gains for reliable multivariate probabilistic forecasting.

Abstract

The goal of probabilistic prediction is to issue predictive distributions that are as informative as possible, subject to being calibrated. Despite substantial progress in the univariate setting, achieving multivariate calibration remains challenging. Recent work has introduced pre-rank functions, scalar projections of multivariate forecasts and observations, as flexible diagnostics for assessing specific aspects of multivariate calibration, but their use has largely been limited to post-hoc evaluation. We propose a regularization-based calibration method that enforces multivariate calibration during training of multivariate distributional regression models using pre-rank functions. We further introduce a novel PCA-based pre-rank that projects predictions onto principal directions of the predictive distribution. Through simulation studies and experiments on 18 real-world multi-output regression datasets, we show that the proposed approach substantially improves multivariate pre-rank calibration without compromising predictive accuracy, and that the PCA pre-rank reveals dependence-structure misspecifications that are not detected by existing pre-ranks.
Paper Structure (46 sections, 1 theorem, 33 equations, 20 figures, 10 tables, 2 algorithms)

This paper contains 46 sections, 1 theorem, 33 equations, 20 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Proper scoring rules
  4. Univariate calibration
  5. Probabilistic Calibration Error (PCE).
  6. Multivariate calibration
  7. Assessing calibration via pre-rank functions.
  8. A New Regularization-based Multivariate Calibration Method
  9. Differentiable Regularizer.
  10. Different Pre-rank Functions.
  11. A New PCA-based Pre-rank
  12. Miscalibration detection with the PCA Pre-rank.
  13. Experiments
  14. Simulation study
  15. Simulation 1 (Multivariate Gaussian).
  16. ...and 31 more sections

Key Result

Proposition 2.1

For every fixed $x \in \mathbb{R}^L$, the function $y \mapsto \rho_2(x, y)$ must be a strictly monotonic bijective transformation of $y \mapsto \rho_1(x, y)$. That is, there exists a strictly increasing or decreasing bijection $h_x$ such that for all $y \in \mathbb{R}^D$,

Figures (20)

  • Figure 1: Marginal calibration is insufficient for multivariate predictive distributions. (Bottom) The model is well-calibrated marginally (per-dimension densities and PIT curves) yet fails to capture dependence, as shown by the mismatch in sample geometry and the strongly miscalibrated Dependency PIT plot. (Top) PCA-based diagnostics reveal miscalibration in projected subspaces, highlighting that calibration assessed only marginally can miss joint errors.
  • Figure 2: PIT histograms for mean, Marginal and PCA pre-rank under different examples of mis-specifications for a multivariate gaussian distribution. While the well-specified model exhibits an approximately uniform PIT, deviations from uniformity reveal errors in the distribution. In this example, the PCA pre-rank detects all considered forms of mis-specification. A full comparison across all pre-ranks is reported in Appendix \ref{['fig:Multivariate_Gaussian']}--\ref{['fig:Gaussian_Fields']}.
  • Figure 3: PCE values with respect to (a) Marginal (b) Scale (c) Dependency and (d) PCA pre-ranks averaged over five runs across 18 benchmark datasets using the MIX-NLL baseline. Blue bars indicate reference PCE values from a simulated perfectly calibrated model (defined as the oracle predictor satisfying $\hat{F}_{Y \mid X} = F_{Y \mid X}$).
  • Figure 4: Performance on benchmark of 18 datasets. Orange: MIX-NLL (no regularization). Blue: MIX-NLL+PCE-KDE (our proposed model). Metrics are calculated across seven pre-rank functions, and averaged over five runs. In subplots (b) and (c), the “None” box refers to the unregularized MIX-NLL trained without pre-rank.
  • Figure 5: Reliability plots on the households dataset using the MIX-NLL+PCE-KDE model after pre-rank regularization. The top row shows calibration curves evaluated with respect to different pre-ranks, while the bottom row reports the corresponding marginal calibration curves.
  • ...and 15 more figures

Theorems & Definitions (4)

  • Definition 2.1: PIT calibration
  • Definition 2.2: Pre-rank function
  • Definition 2.3: Calibration with respect to a pre-rank
  • Proposition 2.1