Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration

Siguang Huang, Yunli Wang, Lili Mou, Huayue Zhang, Han Zhu, Chuan Yu, Bo Zheng

TL;DR

This work tackles the need for calibrated probability estimates in domains where decision-making relies on true event probabilities. It introduces Multiple Boosting Calibration Trees (MBCT), a feature-aware binning framework that uses multiple boosting trees to partition data by input features and outputs, with a per-leaf linear calibrator to achieve individual calibration $h(f(x))=kx$ and potential non-monotonicity. To guide partitioning and evaluation, the authors propose the multi-view calibration error (MVCE), which aggregates calibration errors across multiple partition schemes. Empirical results on industrial CACTRDC data and public Porto Seguro and Avazu datasets show that MBCT improves calibration (MVCE) and order accuracy (AUC), and online deployment in an advertising system demonstrates meaningful gains in CTR and eCPM, underscoring practical value for uncertainty calibration in real systems.

Abstract

Most machine learning classifiers only concern classification accuracy, while certain applications (such as medical diagnosis, meteorological forecasting, and computation advertising) require the model to predict the true probability, known as a calibrated estimate. In previous work, researchers have developed several calibration methods to post-process the outputs of a predictor to obtain calibrated values, such as binning and scaling methods. Compared with scaling, binning methods are shown to have distribution-free theoretical guarantees, which motivates us to prefer binning methods for calibration. However, we notice that existing binning methods have several drawbacks: (a) the binning scheme only considers the original prediction values, thus limiting the calibration performance; and (b) the binning approach is non-individual, mapping multiple samples in a bin to the same value, and thus is not suitable for order-sensitive applications. In this paper, we propose a feature-aware binning framework, called Multiple Boosting Calibration Trees (MBCT), along with a multi-view calibration loss to tackle the above issues. Our MBCT optimizes the binning scheme by the tree structures of features, and adopts a linear function in a tree node to achieve individual calibration. Our MBCT is non-monotonic, and has the potential to improve order accuracy, due to its learnable binning scheme and the individual calibration. We conduct comprehensive experiments on three datasets in different fields. Results show that our method outperforms all competing models in terms of both calibration error and order accuracy. We also conduct simulation experiments, justifying that the proposed multi-view calibration loss is a better metric in modeling calibration error.

MBCT: Tree-Based Feature-Aware Binning for Individual Uncertainty Calibration

TL;DR

This work tackles the need for calibrated probability estimates in domains where decision-making relies on true event probabilities. It introduces Multiple Boosting Calibration Trees (MBCT), a feature-aware binning framework that uses multiple boosting trees to partition data by input features and outputs, with a per-leaf linear calibrator to achieve individual calibration and potential non-monotonicity. To guide partitioning and evaluation, the authors propose the multi-view calibration error (MVCE), which aggregates calibration errors across multiple partition schemes. Empirical results on industrial CACTRDC data and public Porto Seguro and Avazu datasets show that MBCT improves calibration (MVCE) and order accuracy (AUC), and online deployment in an advertising system demonstrates meaningful gains in CTR and eCPM, underscoring practical value for uncertainty calibration in real systems.

Abstract

Most machine learning classifiers only concern classification accuracy, while certain applications (such as medical diagnosis, meteorological forecasting, and computation advertising) require the model to predict the true probability, known as a calibrated estimate. In previous work, researchers have developed several calibration methods to post-process the outputs of a predictor to obtain calibrated values, such as binning and scaling methods. Compared with scaling, binning methods are shown to have distribution-free theoretical guarantees, which motivates us to prefer binning methods for calibration. However, we notice that existing binning methods have several drawbacks: (a) the binning scheme only considers the original prediction values, thus limiting the calibration performance; and (b) the binning approach is non-individual, mapping multiple samples in a bin to the same value, and thus is not suitable for order-sensitive applications. In this paper, we propose a feature-aware binning framework, called Multiple Boosting Calibration Trees (MBCT), along with a multi-view calibration loss to tackle the above issues. Our MBCT optimizes the binning scheme by the tree structures of features, and adopts a linear function in a tree node to achieve individual calibration. Our MBCT is non-monotonic, and has the potential to improve order accuracy, due to its learnable binning scheme and the individual calibration. We conduct comprehensive experiments on three datasets in different fields. Results show that our method outperforms all competing models in terms of both calibration error and order accuracy. We also conduct simulation experiments, justifying that the proposed multi-view calibration loss is a better metric in modeling calibration error.
Paper Structure (22 sections, 1 theorem, 17 equations, 15 figures, 3 tables)

This paper contains 22 sections, 1 theorem, 17 equations, 15 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Approach
  5. Feature-Aware Binning
  6. Individual and Monotonic Calibration
  7. Multiple Boosting Calibration Trees
  8. Experiments
  9. Experimental Setup
  10. Competing Methods
  11. Analysis of the Metrics for Calibration Error
  12. Main Results
  13. In-Depth Analysis
  14. The Impact of Calibration on Real-world Applications
  15. Conclusion
  16. ...and 7 more sections

Key Result

Theorem 1

Let $D_b$ denote the set of $D$ samples that fall into the partition $b$. For any $\alpha\in (0,1)$, partition $b$, predictor $f$ and binning-based calibrator $h$, we have with probability at least $1-\alpha$, where $\hat{y}_b=\frac{1}{c_b}\sum_{i:X_i\in D_b} Y_i$, $\hat{V}_b=\frac{1}{c_b}\sum_{i:X_i\in D_b} (Y_i-\hat{y}_b)^2$, $c_b$ is the number of samples in the $b$th bin, and $B$ is the numbe

Figures (15)

  • Figure 1: An illustration of our Multiple Boosting Calibration Trees (MBCT). The left-hand side shows an example of a binning tree. MBCT may have multiple boosting trees to minimize our MVCE loss. The right-hand side shows several key elements and concepts in our MBCT. Consider a node $i$ with corresponding data $D_i$. It selects a feature $\Phi_*$ by MVCE, and splits the node by the feature values $\varphi_*^1, \varphi_*^2,\varphi_*^3$. For each child node, a linear function $g_j(x)=k_jx$ is fitted for calibration.
  • Figure 2: Main simulation results of ECE, $\text{ECE}_{\text{sweep}}$ and MVCE. The bin numbers of ECE and MVCE are set as 32 in this experiment. The bin number of $\text{ECE}_{\text{sweep}}$ is fixed and determined by itself.
  • Figure 3: $\hat{E}_{bias}$ of MVCE and ECE under different numbers of bins and samples.
  • Figure 4: The difference of MVCE scores in comparison with our MBCT. Natually, our approach yields a value of 0. All competing models have a positive value, showing that our approach is the best.
  • Figure 5: Ablation study of the training loss (in comparison with MVCE training).
  • ...and 10 more figures

Theorems & Definitions (5)

  • Definition 1: Approximate Grouped Calibration
  • Definition 2: Asymptotic Grouped Calibration
  • Theorem 1
  • Definition 3
  • Definition 4