Conformalized Selective Regression

TL;DR

This paper proposes a novel approach to selective regression by leveraging conformal prediction, which provides grounded confidence measures for individual predictions based on model-specific biases, and proposes a standardized evaluation framework to allow proper comparison of selective regression approaches.

Abstract

Should prediction models always deliver a prediction? In the pursuit of maximum predictive performance, critical considerations of reliability and fairness are often overshadowed, particularly when it comes to the role of uncertainty. Selective regression, also known as the "reject option," allows models to abstain from predictions in cases of considerable uncertainty. Initially proposed seven decades ago, approaches to selective regression have mostly focused on distribution-based proxies for measuring uncertainty, particularly conditional variance. However, this focus neglects the significant influence of model-specific biases on a model's performance. In this paper, we propose a novel approach to selective regression by leveraging conformal prediction, which provides grounded confidence measures for individual predictions based on model-specific biases. In addition, we propose a standardized evaluation framework to allow proper comparison of selective regression approaches. Via an extensive experimental approach, we demonstrate how our proposed approach, conformalized selective regression, demonstrates an advantage over multiple state-of-the-art baselines.

Figures (3)

• Figure 1: The dilemma between coverage (proportion of predictions delivered) and error for two hypothetical regression models. Although Model 2 dominates Model 1 in most coverage levels, the goal is finding the best trade-off point between predictive error and coverage, minimizing the former while maximizing the latter, as shown in the right-side plot.
• Figure 2: Comparison of simulated models based on coverage and error, including Euclidean distances from the ideal zero error point and full coverage. The plots illustrate that while a model might have a smaller AUC, it can still offer a more optimal trade-off between accuracy and coverage.
• Figure 3: Model Comparison on Normalized Mean Squared Error (nMSE) vs. Coverage with Random Forest Regressor. This figure compares selective regression models across various datasets. Dashed and solid lines represent different modeling approaches, including CSR variants. Marks indicate optimal performance points - the best balance between prediction accuracy and coverage