Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Loss-Controlling Calibration for Predictive Models

Di Wang, Junzhi Shi, Pingping Wang, Shuo Zhuang, Hongyue Li

TL;DR

This work addresses the limitation of existing conformal methods that require set predictors and monotone losses by introducing loss-controlling calibration (LCC), a framework that enables general loss-controlling predictions under exchangeability. LCC leverages exchangeability-preserving transformations and a predefined searching function to select a predictor $F_{\lambda}$ that meets a loss bound with finite-sample, distribution-free guarantees, even when losses are non-monotone. The method generalizes conformal loss-controlling prediction (CLCP) to arbitrary predictors and losses, and is validated through extensive experiments on selective regression (single and multi-target) and high-impact weather forecasting, demonstrating effective loss control and practical predictive efficiency. This broadens the applicability of conformal risk control to diverse, risk-sensitive applications beyond set-valued predictions.

Abstract

We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data, which extends our recently proposed conformal loss-controlling prediction for more general cases. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors, and the loss function can be any measurable function without the monotone assumption. To control the loss values in an efficient way, we introduce transformations preserving exchangeability to prove finite-sample controlling guarantee when the test label is obtained, and then develop an approximation approach to construct predictors. The transformations can be built on any predefined function, which include using optimization algorithms for parameter searching. This approach is a natural extension of conformal loss-controlling prediction, since it can be reduced to the latter when the set predictors have the nesting property and the loss functions are monotone. Our proposed method is applied to selective regression and high-impact weather forecasting problems, which demonstrates its effectiveness for general loss-controlling prediction.

Loss-Controlling Calibration for Predictive Models

TL;DR

This work addresses the limitation of existing conformal methods that require set predictors and monotone losses by introducing loss-controlling calibration (LCC), a framework that enables general loss-controlling predictions under exchangeability. LCC leverages exchangeability-preserving transformations and a predefined searching function to select a predictor that meets a loss bound with finite-sample, distribution-free guarantees, even when losses are non-monotone. The method generalizes conformal loss-controlling prediction (CLCP) to arbitrary predictors and losses, and is validated through extensive experiments on selective regression (single and multi-target) and high-impact weather forecasting, demonstrating effective loss control and practical predictive efficiency. This broadens the applicability of conformal risk control to diverse, risk-sensitive applications beyond set-valued predictions.

Abstract

We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data, which extends our recently proposed conformal loss-controlling prediction for more general cases. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors, and the loss function can be any measurable function without the monotone assumption. To control the loss values in an efficient way, we introduce transformations preserving exchangeability to prove finite-sample controlling guarantee when the test label is obtained, and then develop an approximation approach to construct predictors. The transformations can be built on any predefined function, which include using optimization algorithms for parameter searching. This approach is a natural extension of conformal loss-controlling prediction, since it can be reduced to the latter when the set predictors have the nesting property and the loss functions are monotone. Our proposed method is applied to selective regression and high-impact weather forecasting problems, which demonstrates its effectiveness for general loss-controlling prediction.
Paper Structure (13 sections, 2 theorems, 37 equations, 7 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 2 theorems, 37 equations, 7 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Inductive Conformal Prediction and Conformal Loss-Controlling Prediction
  3. Inductive Conformal Prediction
  4. Conformal Loss-Controlling Prediction
  5. Loss-Controlling Calibration and Its Theoretical Analysis
  6. Loss-Controlling Calibration
  7. Theoretical Analysis of Loss-Controlling Calibration
  8. Controlling Multiple Losses
  9. Experiments
  10. LCC for Selective Regression with Single Target
  11. LCC for Selective Regression with Multiple Targets
  12. LCC for high-impact weather forecasting
  13. Conclusion

Key Result

Theorem 1

Suppose $\{(X_i, Y_i)\}_{i = 1}^{n+1}$ are $n+1$ data drawn exchangeably from $P_{XY}$ on $\mathcal{X} \times \mathcal{Y}$, $F_{\lambda}: \mathcal{X} \rightarrow \mathcal{Y}'$ is a function with the parameter $\lambda$ taking values from a discrete set $\Lambda$ , $L: \mathcal{Y} \times \mathcal{Y}' then for any $\delta \in (\frac{1}{n+1},1)$, we have where $\hat{\lambda}$ is defined as formula (

Figures (7)

  • Figure 1: Frequencies of the prediction losses being greater than $\alpha$ vs. $\delta = 0.1, 0.15, 0.2$ on test data for selective single-target regression. The first row and the second row correspond to RF and ERT respectively. Different columns represent different $\alpha$. The bars are all near or below $\delta$, indicating the controlling guarantee of LCC empirically.
  • Figure 2: Miscoverage of selective predictions vs. $\delta = 0.1, 0.15, 0.2$ on test data for selective single-target regression. The first row and the second row correspond to RF and ERT respectively. Different columns represent different $\alpha$. Tuning $\alpha$ and $\delta$ can change Miscoverage, which indicates the trade-off between loss level and informational efficiency.
  • Figure 3: Frequencies of the maximum prediction losses being greater than $\alpha$ vs. $\delta = 0.1, 0.15, 0.2$ on test data for selective multi-target regression. The first row and the second row correspond to RF and ERT respectively. Different columns represent different $\alpha$. The bars are all near or below $\delta$, indicating the controlling guarantee based on formula (\ref{['formula_11']}) empirically.
  • Figure 4: Mean Miscoverage of selective predictions vs. $\delta = 0.1, 0.15, 0.2$ on test data for selective multi-target regression. The first row and the second row correspond to RF and ERT respectively. Different columns represent different $\alpha$. Tuning $\alpha$ and $\delta$ can change Mean Miscoverage, which indicates the trade-off between loss level and informational efficiency.
  • Figure 5: Frequencies of the prediction losses being greater than $\alpha$ for different $\delta$ and $\alpha$ on test data of HighTemp and LowTemp datasets. All bars being near or below the preset $\delta$ confirms the controlling guarantee of LCC empirically.
  • ...and 2 more figures

Theorems & Definitions (5)

  • Definition 1
  • Theorem 1
  • proof
  • Corollary 1
  • proof