A generalized approach to label shift: the Conditional Probability Shift Model

TL;DR

This paper introduces Conditional Probability Shift (CPS) as a generalization of distribution shift that captures changes in the conditional label distribution $p(y|z)$ while preserving $p(x|y,z)$. It proposes the Conditional Probability Shift Model (CPSM), which models $p(y|z)$ and $q(y|z)$ via multinomial regression and uses an Expectation-Maximization (EM) procedure to estimate target-domain posteriors $q(y|x,z)$ from unlabeled target data. A key theoretical result shows how $q(y|x,z)$ relates to the source posterior $p(y|x,z)$ and to priors through $p(y|z)$ and $q(y|z)$, enabling Bayes-rule-based classification without needing $p(z)$ and $q(z)$. Empirical evaluation on synthetic data and a MIMIC medical-case study demonstrates that CPSM achieves superior balanced accuracy and lower approximation error than LS- and SJS-based methods, particularly when the conditional distribution shifts but the class priors remain unchanged, with practical runtime trade-offs noted against baselines.

Abstract

In many practical applications of machine learning, a discrepancy often arises between a source distribution from which labeled training examples are drawn and a target distribution for which only unlabeled data is observed. Traditionally, two main scenarios have been considered to address this issue: covariate shift (CS), where only the marginal distribution of features changes, and label shift (LS), which involves a change in the class variable's prior distribution. However, these frameworks do not encompass all forms of distributional shift. This paper introduces a new setting, Conditional Probability Shift (CPS), which captures the case when the conditional distribution of the class variable given some specific features changes while the distribution of remaining features given the specific features and the class is preserved. For this scenario we present the Conditional Probability Shift Model (CPSM) based on modeling the class variable's conditional probabilities using multinomial regression. Since the class variable is not observed for the target data, the parameters of the multinomial model for its distribution are estimated using the Expectation-Maximization algorithm. The proposed method is generic and can be combined with any probabilistic classifier. The effectiveness of CPSM is demonstrated through experiments on synthetic datasets and a case study using the MIMIC medical database, revealing its superior balanced classification accuracy on the target data compared to existing methods, particularly in situations situations of conditional distribution shift and no apriori distribution shift, which are not detected by LS-based methods.

Figures (5)

• Figure 1: Example of a shift in the conditional distribution of a class variable denoting the occurrence of a disease given the age (1st row) : $P(Disease|Age)\neq Q(Disease|Age)$. The prior distribution of the disease is the same for the target and source distributions $P(Disease)=Q(Disease)$ (no label shift). The empirical conditional distribution of the feature (intensity of symptoms) given disease and age are approximately the same for source and target data up to estimation error (2nd row).
• Figure 2: Balanced Accuracy for artificial data 1, for $p(y=1)=0.05$ and varying class prior for target data $q(y=1)=0.05,0.3,0.5,0.8$.
• Figure 3: Balanced Accuracy for artificial data 2, for $p(y=1)=0.05$ and varying class prior for target data $q(y=1)=0.05,0.3,0.5,0.8$.
• Figure 4: Balanced Accuracy and Approximation Error wrt sample size for artificial data 1, $p(y=1)=0.05$ and $k=5$. The class prior for the target data is $q(y=1)=0.5$ (top panels) and $q(y=1)=0.8$ (bottom panels).
• Figure 5: Balanced Accuracy and Approximation Error wrt sample size for artificial data 2, $p(y=1)=0.05$ and $k=5$. The class prior for the target data is $q(y=1)=0.5$ (top panels) and $q(y=1)=0.8$ (bottom panels).

Theorems & Definitions (6)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof