Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibration and Discrimination Optimization Using Clusters of Learned Representation

Tomer Lavi, Bracha Shapira, Nadav Rappoport

TL;DR

This work tackles calibration reliability in high-stakes predictions by recognizing that subpopulations exhibit different calibration behavior. It introduces Clustered Calibration (CCL), an ensemble of per-cluster calibrators guided by clusters in a learned representation, and Clustered Calibration Error (CECE) as a model-selection metric that ties calibration to discrimination. Empirically, CCL improves calibration across diverse calibration methods (e.g., achieving improvements from baseline $ECE$ values up to 100%) and, together with CECE, enables better discrimination as measured by $AUC$ and accuracy. The approach is generic and scalable to clinical decision support and other critical domains, offering interpretable subpopulation calibration and a principled model-selection criterion.

Abstract

Machine learning models are essential for decision-making and risk assessment, requiring highly reliable predictions in terms of both discrimination and calibration. While calibration often receives less attention, it is crucial for critical decisions, such as those in clinical predictions. We introduce a novel calibration pipeline that leverages an ensemble of calibration functions trained on clusters of learned representations of the input samples to enhance overall calibration. This approach not only improves the calibration score of various methods from 82.28% up to 100% but also introduces a unique matching metric that ensures model selection optimizes both discrimination and calibration. Our generic scheme adapts to any underlying representation, clustering, calibration methods and metric, offering flexibility and superior performance across commonly used calibration methods.

Calibration and Discrimination Optimization Using Clusters of Learned Representation

TL;DR

This work tackles calibration reliability in high-stakes predictions by recognizing that subpopulations exhibit different calibration behavior. It introduces Clustered Calibration (CCL), an ensemble of per-cluster calibrators guided by clusters in a learned representation, and Clustered Calibration Error (CECE) as a model-selection metric that ties calibration to discrimination. Empirically, CCL improves calibration across diverse calibration methods (e.g., achieving improvements from baseline values up to 100%) and, together with CECE, enables better discrimination as measured by and accuracy. The approach is generic and scalable to clinical decision support and other critical domains, offering interpretable subpopulation calibration and a principled model-selection criterion.

Abstract

Machine learning models are essential for decision-making and risk assessment, requiring highly reliable predictions in terms of both discrimination and calibration. While calibration often receives less attention, it is crucial for critical decisions, such as those in clinical predictions. We introduce a novel calibration pipeline that leverages an ensemble of calibration functions trained on clusters of learned representations of the input samples to enhance overall calibration. This approach not only improves the calibration score of various methods from 82.28% up to 100% but also introduces a unique matching metric that ensures model selection optimizes both discrimination and calibration. Our generic scheme adapts to any underlying representation, clustering, calibration methods and metric, offering flexibility and superior performance across commonly used calibration methods.
Paper Structure (28 sections, 5 theorems, 7 equations, 4 figures, 4 tables, 2 algorithms)

This paper contains 28 sections, 5 theorems, 7 equations, 4 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Calibration Function
  4. Calibration Error Metrics
  5. Calibration Methods
  6. Multicalibration and Local Calibration
  7. Proposed method
  8. Methods
  9. Input
  10. Data Representation
  11. Clustering
  12. Calibration
  13. The Clustered Calibration Algorithm
  14. Performance Measures
  15. Theoretical Analysis
  16. ...and 13 more sections

Key Result

Lemma 2.1

Let $X$ be a dataset, $C=\{c_1...c_k\}$ set of calibration functions $c$ fit on the $k$ clusters of the calibration dataset, and $CL=\{cl_1...cl_k\}$ a partition of $X$ to $k$ clusters. Then ECE of $c_i$, $cl_i$ for each cluster is upper bound by the calibration error of $c$ on that cluster: $\foral

Figures (4)

  • Figure 1: A binary classification model calibration comparison. The X axis is a division of the data to ten bins sorted by the output values, and the Y axis is the mean probability of the predicted and observed values. The right diagram displays the calibrated outputs of the left diagram.
  • Figure 2: The architecture of the proposed calibration method CCL.
  • Figure 3: A t-SNE visualization of the partition of the WiDS data to 15 clusters using K-means. Left: clusters of the original data points. Middle: top 15% important features. Right: SHAP values.
  • Figure 4: Rejection method error rate comparison.

Theorems & Definitions (8)

  • Lemma 2.1
  • proof
  • Corollary 2.2
  • Lemma 2.3
  • proof
  • Lemma 2.4
  • proof
  • Theorem 2.5