Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization

Naomi Desobry, Elnura Zhalieva, Souhaib Ben Taieb

TL;DR

This work tackles the challenge of calibrating multivariate probabilistic regression by introducing a differentiable pre-rank regularization framework that enforces calibration with respect to selected projection-based diagnostics. It unifies marginal and multivariate calibration through a differentiable penalty, and adds a novel PCA-based pre-rank to capture calibration along directions of maximal predictive variance while enabling dimensionality reduction. Empirically, a MIX-NLL baseline is shown to be systematically miscalibrated across multiple pre-ranks on 18 real-world datasets; applying the proposed regularizers yields substantial calibration improvements without sacrificing predictive accuracy, with PCA-based variants offering scalable performance. The proposed framework provides a practical, generalizable approach to enforce multivariate calibration during training for arbitrary probabilistic predictors and pre-rank choices, enhancing the reliability of multivariate uncertainty quantification in diverse domains.

Abstract

Probabilistic models must be well calibrated to support reliable decision-making. While calibration in single-output regression is well studied, defining and achieving multivariate calibration in multi-output regression remains considerably more challenging. The existing literature on multivariate calibration primarily focuses on diagnostic tools based on pre-rank functions, which are projections that reduce multivariate prediction-observation pairs to univariate summaries to detect specific types of miscalibration. In this work, we go beyond diagnostics and introduce a general regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions. This framework encompasses existing approaches such as highest density region calibration and copula calibration. Our method enforces calibration by penalizing deviations of the projected probability integral transforms (PITs) from the uniform distribution, and can be added as a regularization term to the loss function of any probabilistic predictor. Specifically, we propose a regularization loss that jointly enforces both marginal and multivariate pre-rank calibration. We also introduce a new PCA-based pre-rank that captures calibration along directions of maximal variance in the predictive distribution, while also enabling dimensionality reduction. Across 18 real-world multi-output regression datasets, we show that unregularized models are consistently miscalibrated, and that our methods significantly improve calibration across all pre-rank functions without sacrificing predictive accuracy.

Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization

TL;DR

This work tackles the challenge of calibrating multivariate probabilistic regression by introducing a differentiable pre-rank regularization framework that enforces calibration with respect to selected projection-based diagnostics. It unifies marginal and multivariate calibration through a differentiable penalty, and adds a novel PCA-based pre-rank to capture calibration along directions of maximal predictive variance while enabling dimensionality reduction. Empirically, a MIX-NLL baseline is shown to be systematically miscalibrated across multiple pre-ranks on 18 real-world datasets; applying the proposed regularizers yields substantial calibration improvements without sacrificing predictive accuracy, with PCA-based variants offering scalable performance. The proposed framework provides a practical, generalizable approach to enforce multivariate calibration during training for arbitrary probabilistic predictors and pre-rank choices, enhancing the reliability of multivariate uncertainty quantification in diverse domains.

Abstract

Probabilistic models must be well calibrated to support reliable decision-making. While calibration in single-output regression is well studied, defining and achieving multivariate calibration in multi-output regression remains considerably more challenging. The existing literature on multivariate calibration primarily focuses on diagnostic tools based on pre-rank functions, which are projections that reduce multivariate prediction-observation pairs to univariate summaries to detect specific types of miscalibration. In this work, we go beyond diagnostics and introduce a general regularization framework to enforce multivariate calibration during training for arbitrary pre-rank functions. This framework encompasses existing approaches such as highest density region calibration and copula calibration. Our method enforces calibration by penalizing deviations of the projected probability integral transforms (PITs) from the uniform distribution, and can be added as a regularization term to the loss function of any probabilistic predictor. Specifically, we propose a regularization loss that jointly enforces both marginal and multivariate pre-rank calibration. We also introduce a new PCA-based pre-rank that captures calibration along directions of maximal variance in the predictive distribution, while also enabling dimensionality reduction. Across 18 real-world multi-output regression datasets, we show that unregularized models are consistently miscalibrated, and that our methods significantly improve calibration across all pre-rank functions without sacrificing predictive accuracy.
Paper Structure (28 sections, 2 theorems, 24 equations, 22 figures, 10 tables)

This paper contains 28 sections, 2 theorems, 24 equations, 22 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Regularization for univariate calibration.
  4. Related Work
  5. Multivariate calibration with pre-ranks
  6. An Experimental Study of Multivariate Calibration
  7. Benchmark Datasets.
  8. Neural probabilistic regression model.
  9. Results.
  10. A Pre-rank Regularization Framework
  11. Pre-Rank calibration via Regularization.
  12. Marginal and Pre-rank Calibration.
  13. A PCA-Based Pre-rank and Regularizer.
  14. Equivalence of Pre-Rank Calibration.
  15. Experiments
  16. ...and 13 more sections

Key Result

Proposition 1

For every fixed $x \in \mathbb{R}^L$, the function $y \mapsto \rho_2(x, y)$ must be a strictly monotonic bijective transformation of $y \mapsto \rho_1(x, y)$. That is, there exists a strictly increasing or decreasing bijection $h_x$ such that for all $y \in \mathbb{R}^D$,

Figures (22)

  • Figure 1: PCE values with respect to (a) Location (b) HDR and (c) Copula pre-ranks averaged over five runs across 18 benchmark datasets using the MIX-NLL baseline. Blue bars indicate reference PCE values from a simulated perfectly calibrated model.
  • Figure 2: Performance on 18 real multivariate benchmark datasets. Orange: MIX-NLL (no regularization). Blue: MIX-NLL+PCE-KDE (proposed). Metrics are calculated across seven pre-rank functions, and averaged over five runs. In subplots (b) and (c), the “None” box refers to the unregularized MIX-NLL trained without pre-rank.
  • Figure 3: Reliability plots on wage dataset using MIX-NLL + PCE-KDE on prerank. Top row: calibration curves with respect to (a) Location, (b) HDR, and (c) Copula preranks. Bottom row: corresponding marginal calibration curves.
  • Figure 4: Reliability plots on wage dataset using MIX-NLL + PCE-KDE on marginal+pre-rank. Top row: calibration curves with respect to (a) Location, (b) HDR, and (c) Copula pre-ranks. Bottom row: corresponding marginal calibration curves.
  • Figure 5: PCE values with respect to (a) Marginal (b) Scale (c) Dependency and (d) PCA pre-ranks averaged over five runs across 18 benchmark datasets using the MIX-NLL baseline. Blue bars indicate reference PCE values from a simulated perfectly calibrated model.
  • ...and 17 more figures

Theorems & Definitions (4)

  • Definition 1
  • Definition 2
  • Proposition 1
  • Proposition 2