Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Label Shift Estimation With Incremental Prior Update

Yunrui Zhang, Gustavo Batista, Salil S. Kanhere

Abstract

An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution $p_t(y)$ in the testing set, assuming the likelihood $p(x|y)$ does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an expectation-maximization algorithm. We aim to incrementally update the prior on each sample, adjusting each posterior for more accurate label shift estimation. The proposed method is based on intuitive assumptions on classifiers that are generally true for modern probabilistic classifiers. The proposed method relies on a weaker notion of calibration compared to other methods. As a post-hoc approach for label shift estimation, the proposed method is versatile and can be applied to any black-box probabilistic classifier. Experiments on CIFAR-10 and MNIST show that the proposed method consistently outperforms the current state-of-the-art maximum likelihood-based methods under different calibrations and varying intensities of label shift.

Label Shift Estimation With Incremental Prior Update

Abstract

An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution in the testing set, assuming the likelihood does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an expectation-maximization algorithm. We aim to incrementally update the prior on each sample, adjusting each posterior for more accurate label shift estimation. The proposed method is based on intuitive assumptions on classifiers that are generally true for modern probabilistic classifiers. The proposed method relies on a weaker notion of calibration compared to other methods. As a post-hoc approach for label shift estimation, the proposed method is versatile and can be applied to any black-box probabilistic classifier. Experiments on CIFAR-10 and MNIST show that the proposed method consistently outperforms the current state-of-the-art maximum likelihood-based methods under different calibrations and varying intensities of label shift.

Paper Structure

This paper contains 12 sections, 1 theorem, 13 equations, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Setup
  3. Related Work
  4. LEIP
  5. Selecting $\tau$
  6. The Effect of Calibration
  7. Experiments
  8. Conclusion
  9. Appendix
  10. Derivation of Equation \ref{['LESA:1']} from Bayes' Theorem
  11. Dirichlet Shift examples
  12. Comparison of different approaches of calculating weight for EM and LEIP

Key Result

Corollary 1

For a classifier $\mathcal{F}$ with a probability confusion matrix $C_{\hat{y}, y}$ estimated from the validation set, where each column $C_{\cdot, y}$ sums to one. The minimum accuracy when predicting on a set of samples with an unknown label distribution is given by the minimum recall that is $\mi

Figures (1)

  • Figure 1: Comparison of CC, RLLS, EM and LEIP under Dirichlet shift for a wide range of $\alpha$. Numbers reported are on the scale of $10^{-3}$; for each $\alpha$, the MSE reported are average across 50 runs. Validation set size 2500 for CIFAR10 and 1500 for MNIST.

Theorems & Definitions (5)

  • Definition 1
  • Definition 2
  • Definition 3
  • Corollary 1
  • Definition 4