Open Set Label Shift with Test Time Out-of-Distribution Reference

TL;DR

This work tackles Open Set Label Shift (OSLS), where the target domain contains an OOD class and the ID data distribution shifts from the source while the conditional data model $p(x|y)$ is preserved. It introduces a three-stage, test-time OSLS-EM framework that uses a source ID classifier and an ID/OOD detector, plus a test-time OOD reference dataset, to estimate the target ID distribution ${p_t(y)=\boldsymbol{\pi}}$ and target ID data ratio ${p_t(b=1)=\rho_t}$, and to correct the source classifier without retraining. The estimation relies on a convex, reparameterized maximum likelihood objective and EM iterations (MLE/MAP variants), with a corrective step to handle imperfect OOD detectors via a linear rho-t correction; an explicit source data-ratio estimator ${\hat{\rho}}_s$ is provided with concentration bounds. An OOD reference can be pseudo-generated at test time, enabling flexible deployment across datasets like CIFAR-10/100 and ImageNet-200; experiments demonstrate improvements over CSLS baselines in both estimation error and correction accuracy, validating practical OSLS deployment. The approach thus enables robust domain adaptation under open-set shifts without retraining, broadening applicability in real-world deployment where labeled target data and OOD samples are scarce.

Abstract

Open set label shift (OSLS) occurs when label distributions change from a source to a target distribution, and the target distribution has an additional out-of-distribution (OOD) class. In this work, we build estimators for both source and target open set label distributions using a source domain in-distribution (ID) classifier and an ID/OOD classifier. With reasonable assumptions on the ID/OOD classifier, the estimators are assembled into a sequence of three stages: 1) an estimate of the source label distribution of the OOD class, 2) an EM algorithm for Maximum Likelihood estimates (MLE) of the target label distribution, and 3) an estimate of the target label distribution of OOD class under relaxed assumptions on the OOD classifier. The sampling errors of estimates in 1) and 3) are quantified with a concentration inequality. The estimation result allows us to correct the ID classifier trained on the source distribution to the target distribution without retraining. Experiments on a variety of open set label shift settings demonstrate the effectiveness of our model. Our code is available at https://github.com/ChangkunYe/OpenSetLabelShift.

Figures (9)

• Figure 1: Open Set Label Shift (OSLS) set up, where source and target domain have different label distributions $p(y)$ but identical conditional distribution of data given label $p(x|y)$. OSLS extends the Closed Set Label Shift (CSLS) with an extra Out-of-Distribution (OOD) class on the target domain.
• Figure 2: Structure of our proposed Open Set Label Shift estimation and correction method. The target ID label distribution probabilities ${p_t(y=\cdot)=\boldsymbol{\mathbf{\pi}}}$ and ID data ratio ${p_t(b=1)=\rho_t}$ are estimated through three steps: 1) retrieve source ID data ratio ${\rho_s}$ (Sec. \ref{['subsec:rho-s-retrieval']}), 2) estimate target ID data ratio ${\rho_t}$ and target ID label distribution $\boldsymbol{\mathbf{\pi}}$ via an EM algorithm under Assumption \ref{['assume:A1']},\ref{['assume:A2']} (Sec. \ref{['subsec:em']}) and 3) correct the target ID data ratio estimator $\hat{\rho}_t$ when OOD classifier $h(x)$ does not satisfy Assumption. \ref{['assume:A2']}B (Sec. \ref{['subsec:rho-t-correction']}). Based on the estimation result, our correction model constructs a new classifier to classify target domain images. Unlike garg2022domain that retrains the ID/OOD classifier, any OOD classifier proposed in previous OOD detection literature can be used in our model without retraining.
• Figure 3: Graphical model of the Open Set Label Shift setting and our assumptions.$X_s, X_t$ are data for the source and target domain, $Y_s, Y_t$ are the corresponding categorical-valued labels and $B_s, B_t$ are binary values representing ID/OOD data. $\boldsymbol{\mathbf{c}}, \boldsymbol{\mathbf{\pi}}$ are source and target domain label distribution class probabilities. Source domain data $X_s$ is observed with ground truth ID data in $\mathcal{D}^s$ and reference OOD data in $\mathcal{D}^{\textbf{o}}$. $p(x|y)$ are invariant under the label shift assumption. With the help of a reference OOD dataset $\mathcal{D}^{\text{o}}$ at test time, we first estimate $\rho_s, \boldsymbol{\mathbf{c}}$ and then $\rho_t, \boldsymbol{\mathbf{\pi}}$ without retraining. Optional prior distributions are employed on $\rho_t, \boldsymbol{\mathbf{\pi}}$.
• Figure 4: Estimation result comparison of $\hat{\rho}_t^{*}$ by our model (Solid lines), $\hat{\rho_t}$ by our model but without $\rho_t$ correction (§ \ref{['subsec:rho-t-correction']}) (Dashed lines) based on different OOD classifiers and the Ground truth $\rho_t$ (Black, Solid line), on CIFAR10/100 dataset with Dirichlet shift and Near OOD dataset (Tab. \ref{['tab:dataset-setup']}). The estimation result exhibit a linear correlation with the ground truth, which is explained by our analysis in Theorem \ref{['theorem:estrho-linear']} Moreover, our $\rho_t$ correction model is able to adjust the predicted $\hat{\rho}_t$ to $\hat{\rho}_t^{*}$ that is closer to the ground truth. Shaded area are $\pm$ one standard deviation over three independent ID classifiers.
• Figure 5: Estimation result comparison of $\hat{\rho}_t^{*}$ by our model (Solid lines), $\hat{\rho_t}$ by our model but without $\rho_t$ correction (Sec. \ref{['subsec:rho-t-correction']}) (Dashed lines) based on different OOD classifiers and the Ground truth $\rho_t$ (Black, Solid line), on CIFAR10/100 dataset with Dirichlet shift and Near + Far OOD dataset (Tab. \ref{['tab:dataset-setup']}). Shaded area are $\pm$ one standard deviation over corresponding OOD datasets and three independent ID classifiers.

Theorems & Definitions (21)

• Definition 3.1
• Theorem 4.1
• Lemma 4.1
• Theorem 4.2
• Theorem 4.3
• Theorem C.1
• proof
• Lemma C.0
• proof
• Lemma C.0