Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neyman-Pearson Classification under Both Null and Alternative Distributions Shift

Mohammadreza M. Kalan, Yuyang Deng, Eitan J. Neugut, Samory Kpotufe

TL;DR

This work addresses transfer learning for Neyman–Pearson classification under potential shifts in both the source and target class‑conditionals. It proposes a two‑stage adaptive procedure that aligns the source constraint with the target and then refines the model using source class‑1 data, ensuring the target Type‑I constraint while reducing the target Type‑II error when the source is informative and avoiding negative transfer otherwise. Theoretical guarantees are provided via transfer moduli $\phi_{1}^{S\to T}$ and $\phi_{0}^{S\to T}$ that bound the excess risk, alongside a computational framework based on two staged convex programs solved with SGDA, yielding polynomial‑time assurance. Empirical results on climate datasets demonstrate adaptive gains when the source is helpful and robust performance when it is not, underscoring the method's practical value for imbalanced NP transfer learning.

Abstract

We consider the problem of transfer learning in Neyman-Pearson classification, where the objective is to minimize the error w.r.t. a distribution $μ_1$, subject to the constraint that the error w.r.t. a distribution $μ_0$ remains below a prescribed threshold. While transfer learning has been extensively studied in traditional classification, transfer learning in imbalanced classification such as Neyman-Pearson classification has received much less attention. This setting poses unique challenges, as both types of errors must be simultaneously controlled. Existing works address only the case of distribution shift in $μ_1$, whereas in many practical scenarios shifts may occur in both $μ_0$ and $μ_1$. We derive an adaptive procedure that not only guarantees improved Type-I and Type-II errors when the source is informative, but also automatically adapt to situations where the source is uninformative, thereby avoiding negative transfer. In addition to such statistical guarantees, the procedures is efficient, as shown via complementary computational guarantees.

Neyman-Pearson Classification under Both Null and Alternative Distributions Shift

TL;DR

This work addresses transfer learning for Neyman–Pearson classification under potential shifts in both the source and target class‑conditionals. It proposes a two‑stage adaptive procedure that aligns the source constraint with the target and then refines the model using source class‑1 data, ensuring the target Type‑I constraint while reducing the target Type‑II error when the source is informative and avoiding negative transfer otherwise. Theoretical guarantees are provided via transfer moduli and that bound the excess risk, alongside a computational framework based on two staged convex programs solved with SGDA, yielding polynomial‑time assurance. Empirical results on climate datasets demonstrate adaptive gains when the source is helpful and robust performance when it is not, underscoring the method's practical value for imbalanced NP transfer learning.

Abstract

We consider the problem of transfer learning in Neyman-Pearson classification, where the objective is to minimize the error w.r.t. a distribution , subject to the constraint that the error w.r.t. a distribution remains below a prescribed threshold. While transfer learning has been extensively studied in traditional classification, transfer learning in imbalanced classification such as Neyman-Pearson classification has received much less attention. This setting poses unique challenges, as both types of errors must be simultaneously controlled. Existing works address only the case of distribution shift in , whereas in many practical scenarios shifts may occur in both and . We derive an adaptive procedure that not only guarantees improved Type-I and Type-II errors when the source is informative, but also automatically adapt to situations where the source is uninformative, thereby avoiding negative transfer. In addition to such statistical guarantees, the procedures is efficient, as shown via complementary computational guarantees.

Paper Structure

This paper contains 17 sections, 6 theorems, 93 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Setup
  4. Adaptive transfer learning procedure
  5. Main Results
  6. Generalization Error Bounds
  7. Computational Guarantee for the Transfer Learning Procedure
  8. Notations.
  9. Second Stage: Solving the Final Model $\hat{h}$.
  10. Experiments
  11. Expressing Theorem \ref{['theorem_bound']} in Terms of Transfer Exponent
  12. Proof of Theorem \ref{['theorem_bound']}
  13. Proof of Theorem \ref{['thm:opt rate']}
  14. Proof of Theorem \ref{['thm:cpsolver rate']}
  15. Proof of Theorem \ref{['thm:opt rate']}
  16. ...and 2 more sections

Key Result

Theorem 1

Suppose that the hypothesis class $\mathcal{H}$ satisfies Assumptions assump_rademacher and assump-convexity. Moreover, let $\delta > 0$ and $\epsilon_{i,D} = \frac{\tilde{C}}{\sqrt{n_{i,D}}}$ for $i \in \{0,1\}$ and $D \in \{S,T\}$, where $\tilde{C} = 8B_{\mathcal{H}}L + 2C\sqrt{2\log\!\left(\tfrac where $c$ is a universal constant and $\hat{\alpha}_S$ is the empirical threshold defined in (empir

Figures (3)

  • Figure 1: In this figure, we consider one target and two sources that all share the distribution $\mu_1$, while $\mu_0$ differs across them. All distributions are Gaussian with the same variance but different means. The optimal NP classifiers are denoted by $h^*_{T,\alpha}$, $h^*_{S_1,\alpha}$, and $h^*_{S_2,\alpha}$. Moreover, we assume that there are sufficiently many target samples such that $\hat{\mathcal{H}}^*_{\alpha,T}$ does not intersect with $h^*_{S_1,\alpha}$ or $h^*_{S_2,\alpha}$, which implies that $\alpha_{S_2}>\alpha_{S_1}=\alpha$
  • Figure 2: The performance of our algorithm (TLA), along with two baselines---using only source data and only target data---on Climate data yu2023climsim, is evaluated under a Type-I error threshold of $\alpha = 0.1$. In this experiment, we fix $n_{0,T} = n_{1,T} = 40$ and vary the number of source samples $n_{0,S} = n_{1,S}$.
  • Figure 3: The performance of our algorithm (TLA), along with two baselines---using only source data and only target data---on Climate data nasa_power_api, is evaluated under a Type-I error threshold of $\alpha = 0.1$. In this experiment, we fix $n_{0,T} = n_{1,T} = 40$ and vary the number of source samples $n_{0,S} = n_{1,S}$.

Theorems & Definitions (21)

  • Definition 1: Surrogate Loss
  • Definition 2
  • Definition 3: Rademacher Complexity bartlett2002rademacher
  • Definition 4: Transfer Modulus
  • Theorem 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Definition 5
  • ...and 11 more