Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift

Alejandro Moreo

TL;DR

The paper addresses dataset shift by linking calibration, quantification, and classifier accuracy prediction through formal reductions, proving that an oracle for any one task yields solutions to the other two. Building on this theoretical bridge, it introduces cross-domain adaptations and two novel calibration methods, PacCal^σ and DMcal, inspired by quantification approaches to improve robustness under CS and LS. Comprehensive experiments show that adapted methods can rival or surpass dedicated baselines in several settings, with notable successes for TransCal in CS quantification and LasCal in LS calibration, and DMcal offering strong cross-regime calibration. The findings support a unified, cross-disciplinary trajectory for handling distribution shift, highlight the importance of shift-type awareness, and point to promising directions toward multiclass extensions and automated method selection.

Abstract

When the distribution of the data used to train a classifier differs from that of the test data, i.e., under dataset shift, well-established routines for calibrating the decision scores of the classifier, estimating the proportion of positives in a test sample, or estimating the accuracy of the classifier, become particularly challenging. This paper investigates the interconnections among three fundamental problems, calibration, quantification, and classifier accuracy prediction, under dataset shift conditions. Specifically, we prove their equivalence through mutual reduction, i.e., we show that access to an oracle for any one of these tasks enables the resolution of the other two. Based on these proofs, we propose new methods for each problem based on direct adaptations of well-established methods borrowed from the other disciplines. Our results show such methods are often competitive, and sometimes even surpass the performance of dedicated approaches from each discipline. The main goal of this paper is to fostering cross-fertilization among these research areas, encouraging the development of unified approaches and promoting synergies across the fields.

On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift

TL;DR

The paper addresses dataset shift by linking calibration, quantification, and classifier accuracy prediction through formal reductions, proving that an oracle for any one task yields solutions to the other two. Building on this theoretical bridge, it introduces cross-domain adaptations and two novel calibration methods, PacCal^σ and DMcal, inspired by quantification approaches to improve robustness under CS and LS. Comprehensive experiments show that adapted methods can rival or surpass dedicated baselines in several settings, with notable successes for TransCal in CS quantification and LasCal in LS calibration, and DMcal offering strong cross-regime calibration. The findings support a unified, cross-disciplinary trajectory for handling distribution shift, highlight the importance of shift-type awareness, and point to promising directions toward multiclass extensions and automated method selection.

Abstract

When the distribution of the data used to train a classifier differs from that of the test data, i.e., under dataset shift, well-established routines for calibrating the decision scores of the classifier, estimating the proportion of positives in a test sample, or estimating the accuracy of the classifier, become particularly challenging. This paper investigates the interconnections among three fundamental problems, calibration, quantification, and classifier accuracy prediction, under dataset shift conditions. Specifically, we prove their equivalence through mutual reduction, i.e., we show that access to an oracle for any one of these tasks enables the resolution of the other two. Based on these proofs, we propose new methods for each problem based on direct adaptations of well-established methods borrowed from the other disciplines. Our results show such methods are often competitive, and sometimes even surpass the performance of dedicated approaches from each discipline. The main goal of this paper is to fostering cross-fertilization among these research areas, encouraging the development of unified approaches and promoting synergies across the fields.
Paper Structure (34 sections, 6 theorems, 53 equations, 4 figures, 11 tables)

This paper contains 34 sections, 6 theorems, 53 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Calibration
  4. Quantification
  5. Classifier Accuracy Prediction
  6. Preliminaries
  7. Notation
  8. Definitions
  9. Problem Equivalences via Reduction
  10. Access to a Perfect Calibrator $\zeta^*$
  11. Access to a Perfect Quantifier $\rho^*$
  12. Access to a Perfect Estimator of Classifier Accuracy $\alpha^*$
  13. Methods
  14. Direct Adaptations
  15. Original New Methods for Calibration
  16. ...and 19 more sections

Key Result

Lemma 1

$\zeta^* \implies \rho^*$, i.e., from a perfect calibrator we can attain a perfect quantifier.

Figures (4)

  • Figure 1: Graphical explanation of DMcal. First row: The left-most panel shows the class-conditional distributions, represented as histograms of posterior probabilities ($H_{val}^{\ominus}$ in blue, and $H_{val}^{\oplus}$ in orange) modelled on validation data. The central panel displays the test histogram ($H_{te}$). The right-most panel displays the mixture model of $H_{val}^{\ominus}$ and $H_{val}^{\oplus}$ obtained with the mixture parameter $p=0.3$ which yields the closest mixture to the test histogram in terms of HD. Second row: the left-most panel shows a "raw" calibration map computed on the proportion of positive and negative contributions of each bin of the mixture model; the right-most panel shows the corrected calibration map after imposing monotonicity and smoothing the raw calibration map.
  • Figure 2: Calibration error in terms of ECE as a function of shift intensity in CS experiments (left panel) and LS (right panel).
  • Figure 4: Quantification error in terms of AE as a function of shift intensity in CS experiments (left panel) and LS (right panel).
  • Figure 6: Classifier accuracy prediction error in terms of AE as a function of shift intensity in CS experiments (left panel) and LS (right panel).

Theorems & Definitions (21)

  • Definition 3.1
  • Definition 3.2
  • Remark 1
  • Definition 3.3
  • Remark 2
  • Remark 3
  • Definition 3.4
  • Definition 3.5
  • Remark 4
  • Lemma 1
  • ...and 11 more