Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributional regression with reject option

Ahmed Zaoui, Clément Dombry

TL;DR

The paper addresses the challenge of making reliable distributional predictions when uncertainty is high by introducing a reject option for distributional regression with a fixed rejection rate. It derives an optimal rule based on thresholding the entropy of the CRPS, and proposes a semi-supervised plug-in procedure that uses labeled data to estimate the conditional distribution and unlabeled data to calibrate the rejection threshold, achieving distribution-free control of the rejection rate. The authors establish consistency and convergence rates, including non-asymptotic bounds and rate results for distributional KNN, and validate the approach on real datasets showing that better calibration of the entropy threshold improves predictive risk. The work contributes a flexible, practically effective framework for selective distributional prediction with strong theoretical guarantees and empirical support, enabling robust decision-making under uncertainty.

Abstract

Selective prediction, where a model has the option to abstain from making a decision, is crucial for machine learning applications in which mistakes are costly. In this work, we focus on distributional regression and introduce a framework that enables the model to abstain from estimation in situations of high uncertainty. We refer to this approach as distributional regression with reject option, inspired by similar concepts in classification and regression with reject option. We study the scenario where the rejection rate is fixed. We derive a closed-form expression for the optimal rule, which relies on thresholding the entropy function of the Continuous Ranked Probability Score (CRPS). We propose a semi-supervised estimation procedure for the optimal rule, using two datasets: the first, labeled, is used to estimate both the conditional distribution function and the entropy function of the CRPS, while the second, unlabeled, is employed to calibrate the desired rejection rate. Notably, the control of the rejection rate is distribution-free. Under mild conditions, we show that our procedure is asymptotically as effective as the optimal rule, both in terms of error rate and rejection rate. Additionally, we establish rates of convergence for our approach based on distributional k-nearest neighbor. A numerical analysis on real-world datasets demonstrates the strong performance of our procedure

Distributional regression with reject option

TL;DR

The paper addresses the challenge of making reliable distributional predictions when uncertainty is high by introducing a reject option for distributional regression with a fixed rejection rate. It derives an optimal rule based on thresholding the entropy of the CRPS, and proposes a semi-supervised plug-in procedure that uses labeled data to estimate the conditional distribution and unlabeled data to calibrate the rejection threshold, achieving distribution-free control of the rejection rate. The authors establish consistency and convergence rates, including non-asymptotic bounds and rate results for distributional KNN, and validate the approach on real datasets showing that better calibration of the entropy threshold improves predictive risk. The work contributes a flexible, practically effective framework for selective distributional prediction with strong theoretical guarantees and empirical support, enabling robust decision-making under uncertainty.

Abstract

Selective prediction, where a model has the option to abstain from making a decision, is crucial for machine learning applications in which mistakes are costly. In this work, we focus on distributional regression and introduce a framework that enables the model to abstain from estimation in situations of high uncertainty. We refer to this approach as distributional regression with reject option, inspired by similar concepts in classification and regression with reject option. We study the scenario where the rejection rate is fixed. We derive a closed-form expression for the optimal rule, which relies on thresholding the entropy function of the Continuous Ranked Probability Score (CRPS). We propose a semi-supervised estimation procedure for the optimal rule, using two datasets: the first, labeled, is used to estimate both the conditional distribution function and the entropy function of the CRPS, while the second, unlabeled, is employed to calibrate the desired rejection rate. Notably, the control of the rejection rate is distribution-free. Under mild conditions, we show that our procedure is asymptotically as effective as the optimal rule, both in terms of error rate and rejection rate. Additionally, we establish rates of convergence for our approach based on distributional k-nearest neighbor. A numerical analysis on real-world datasets demonstrates the strong performance of our procedure

Paper Structure

This paper contains 16 sections, 17 theorems, 84 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. General framework
  3. The Continuous Ranked Probability Score
  4. Distributional regression with reject option
  5. Fixed rejection rate
  6. Data-driven procedure
  7. Statistical guarantees
  8. Consistency result
  9. Rates of convergence for distributional KNN
  10. Numerical results
  11. Conclusion and perspectives
  12. Acknowledgement
  13. Technical lemmas
  14. Proof of Section \ref{['sec:generalframework']}
  15. Proof of Section \ref{['sec:statisticalguarantees']}
  16. ...and 1 more sections

Key Result

Proposition 1

For any $\lambda > 0$, the optimal predictor is unique (modulo null probability sets) and is defined by

Figures (2)

  • Figure 1: Visual description of the performance of two plug-in $\varepsilon$-predictors on the $\texttt{qsar}$, concrete, and $\texttt{airfoil}$ datasets.
  • Figure 2: $\widehat{{\rm Err}}\left(\hat{\Gamma}_{\lambda}\right)$ and $\hat{r}\left(\hat{\Gamma}_{\lambda}\right)$ vs. $\lambda$.

Theorems & Definitions (29)

  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Remark 1
  • Proposition 5
  • Theorem 1
  • Corollary 1
  • Corollary 2
  • Theorem 2
  • ...and 19 more