Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Normalizing Flows for Conformal Regression

Nicolo Colombo

TL;DR

The framework allows estimating the gap between nominal and empirical conditional validity and is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.

Abstract

Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. The same calibration scheme usually applies to any model and data without modifications. The obtained prediction intervals are valid by construction but could be inefficient, i.e. unnecessarily big, if the prediction errors are not uniformly distributed over the input space. We present a general scheme to localize the intervals by training the calibration process. The standard prediction error is replaced by an optimized distance metric that depends explicitly on the object attributes. Learning the optimal metric is equivalent to training a Normalizing Flow that acts on the joint distribution of the errors and the inputs. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity. The approach is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.

Normalizing Flows for Conformal Regression

TL;DR

The framework allows estimating the gap between nominal and empirical conditional validity and is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.

Abstract

Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. The same calibration scheme usually applies to any model and data without modifications. The obtained prediction intervals are valid by construction but could be inefficient, i.e. unnecessarily big, if the prediction errors are not uniformly distributed over the input space. We present a general scheme to localize the intervals by training the calibration process. The standard prediction error is replaced by an optimized distance metric that depends explicitly on the object attributes. Learning the optimal metric is equivalent to training a Normalizing Flow that acts on the joint distribution of the errors and the inputs. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity. The approach is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.
Paper Structure (23 sections, 6 theorems, 31 equations, 2 figures, 4 tables)

This paper contains 23 sections, 6 theorems, 31 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Outline
  3. An example
  4. Theory
  5. Quantiles
  6. Conformity scores
  7. Normalizing Flows
  8. Validity
  9. Exact Normalizing Flows
  10. Non-exact Normalizing Flows
  11. Implementation
  12. Data
  13. Models
  14. Results
  15. Related work
  16. ...and 8 more sections

Key Result

Lemma 2.1

Let $Z_1, \dots, Z_{N}, Z_{N+1} \in {\mathbb R}$ be a collection of i.i.d. random variables and $Q_Z$ be the $(1-\alpha)$-th sample quantile of $\{ Z_n \}_{n=1}^N$ defined in sample quantile definition. If ties occur with probability 0,

Figures (2)

  • Figure 1: A test sample of conformity scores (black diamonds) and the upper bound of the marginal PIs (black dots) and the adaptive PIs obtained through the ER CP algorithm of papadopoulos2008normalized (blue dots) and the NF approach (red dots). The nominal confidence level is $1-\alpha = 0.9$ for all algorithms.
  • Figure 2: A calibration sample of the original conformity scores (black diamonds) and the scores obtained by transforming them with $b_{ER}$ (blue stars) and $b_{flow}$ (red dots). The solid and dashed lines represent the corresponding $(1 - \alpha)$-th sample quantiles, $1-\alpha = 0.9$.

Theorems & Definitions (6)

  • Lemma 2.1: Quantile Lemma tibshirani2019conformal
  • Lemma 2.3
  • Lemma 2.4
  • Theorem 2.5
  • Corollary 2.6
  • Theorem 2.7