Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neyman-Pearson Multi-class Classification via Cost-sensitive Learning

Ye Tian, Yang Feng

TL;DR

This work tackles multi-class asymmetric classification by linking Neyman-Pearson NP constraints with cost-sensitive (CS) learning through strong duality. It introduces two practical algorithms, NPMC-CX and NPMC-ER, that solve the NP multi-class problem by solving a CS problem in the dual, with theoretical NP oracle properties and conditions for strong duality. The authors provide feasibility and strong duality checking procedures, enabling practitioners to map out the landscape of a given NPMC problem and select target error levels. Empirical results on simulations and real data (e.g., LendingClub) demonstrate that the proposed methods effectively control per-class errors around specified targets while maintaining competitive overall performance, outperforming vanilla classifiers under imbalanced conditions. The work also extends the framework to more general confusion-matrix controls (GNPMC) and discusses extensions, consistency, and practical considerations for broad applicability.

Abstract

Most existing classification methods aim to minimize the overall misclassification error rate. However, in applications such as loan default prediction, different types of errors can have varying consequences. To address this asymmetry issue, two popular paradigms have been developed: the Neyman-Pearson (NP) paradigm and the cost-sensitive (CS) paradigm. Previous studies on the NP paradigm have primarily focused on the binary case, while the multi-class NP problem poses a greater challenge due to its unknown feasibility. In this work, we tackle the multi-class NP problem by establishing a connection with the CS problem via strong duality and propose two algorithms. We extend the concept of NP oracle inequalities, crucial in binary classifications, to NP oracle properties in the multi-class context. Our algorithms satisfy these NP oracle properties under certain conditions. Furthermore, we develop practical algorithms to assess the feasibility and strong duality in multi-class NP problems, which can offer practitioners the landscape of a multi-class NP problem with various target error levels. Simulations and real data studies validate the effectiveness of our algorithms. To our knowledge, this is the first study to address the multi-class NP problem with theoretical guarantees. The proposed algorithms have been implemented in the R package \texttt{npcs}, which is available on CRAN.

Neyman-Pearson Multi-class Classification via Cost-sensitive Learning

TL;DR

This work tackles multi-class asymmetric classification by linking Neyman-Pearson NP constraints with cost-sensitive (CS) learning through strong duality. It introduces two practical algorithms, NPMC-CX and NPMC-ER, that solve the NP multi-class problem by solving a CS problem in the dual, with theoretical NP oracle properties and conditions for strong duality. The authors provide feasibility and strong duality checking procedures, enabling practitioners to map out the landscape of a given NPMC problem and select target error levels. Empirical results on simulations and real data (e.g., LendingClub) demonstrate that the proposed methods effectively control per-class errors around specified targets while maintaining competitive overall performance, outperforming vanilla classifiers under imbalanced conditions. The work also extends the framework to more general confusion-matrix controls (GNPMC) and discusses extensions, consistency, and practical considerations for broad applicability.

Abstract

Most existing classification methods aim to minimize the overall misclassification error rate. However, in applications such as loan default prediction, different types of errors can have varying consequences. To address this asymmetry issue, two popular paradigms have been developed: the Neyman-Pearson (NP) paradigm and the cost-sensitive (CS) paradigm. Previous studies on the NP paradigm have primarily focused on the binary case, while the multi-class NP problem poses a greater challenge due to its unknown feasibility. In this work, we tackle the multi-class NP problem by establishing a connection with the CS problem via strong duality and propose two algorithms. We extend the concept of NP oracle inequalities, crucial in binary classifications, to NP oracle properties in the multi-class context. Our algorithms satisfy these NP oracle properties under certain conditions. Furthermore, we develop practical algorithms to assess the feasibility and strong duality in multi-class NP problems, which can offer practitioners the landscape of a multi-class NP problem with various target error levels. Simulations and real data studies validate the effectiveness of our algorithms. To our knowledge, this is the first study to address the multi-class NP problem with theoretical guarantees. The proposed algorithms have been implemented in the R package \texttt{npcs}, which is available on CRAN.

Paper Structure

This paper contains 70 sections, 24 theorems, 156 equations, 18 figures, 8 algorithms.

Table of Contents

  1. Introduction
  2. Asymmetric classification errors and an example in loan default prediction
  3. Neyman-Pearson paradigm
  4. Cost-sensitive learning
  5. Multi-class NP oracle properties
  6. Organization
  7. Notations
  8. Methodology
  9. The first algorithm: NPMC-CX
  10. The second algorithm: NPMC-ER
  11. Theory
  12. Checking strong duality and feasibility
  13. Analysis on NPMC-CX
  14. Analysis on NPMC-ER
  15. Comparison of NPMC-CX and NPMC-ER from theoretical perspective
  16. ...and 55 more sections

Key Result

Lemma 1

The 3-class NPMC problem eq: NPMC with $X|Y = k \sim N(\bm{\mu}_k, \bm{I}_p)$ for $k = 1, 2, 3$, $\mathcal{A} = \{1,2\}$, and the target levels $\alpha_1$, $\alpha_2 \in [0,1]$, is feasible if and only if where $\Phi^{-1}$ is the inverse CDF function of $N(0, 1)$.

Figures (18)

  • Figure 1: Per-class error rates under each classifier and training sample size setting in simulation. Horizontal lines in corresponding colors mark the target control levels. In some graphs, additional values are displayed in brackets beneath the training sample size, $n$. These values represent the number of instances where the algorithms reported infeasibility during evaluation.
  • Figure 2: Strong duality and feasibility of simulation: ground truth and predicted results.
  • Figure 3: Strong duality and feasibility of NPMC problem for the LendingClub dataset with different target error levels: predicted by Algorithm \ref{['algo: df check CX']} with NPMC-CX-logistic.
  • Figure 4: Strong duality and feasibility of NPMC problem for the LendingClub dataset with different target error levels: predicted by Algorithm \ref{['algo: df check ER']} with NPMC-ER-RF.
  • Figure 5: Per-class error rates and objective function values under each classifier for the NPMC problem on the LendingClub dataset. Horizontal lines in corresponding colors mark the target control levels.
  • ...and 13 more figures

Theorems & Definitions (27)

  • Lemma 1
  • Lemma 2
  • Theorem 1: Sufficient and necessary conditions for NPMC strong duality
  • Corollary 1
  • Proposition 1
  • Remark 1
  • Theorem 2: Multi-class NP oracle properties of NPMC-CX
  • Remark 2
  • Theorem 3: Multi-class NP oracle properties of NPMC-ER
  • Lemma 3
  • ...and 17 more