Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conjunction Subspaces Test for Conformal and Selective Classification

Zengyou He, Zerun Li, Junjie Dong, Xinying Liu, Mudi Jiang, Lianyu Hu

TL;DR

A new classifier is presented, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision.

Abstract

In this paper, we present a new classifier, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision. The null hypothesis is that the test sample has no association with the target class on a randomly chosen subspace, and hence the classification problem can be formulated as a problem of testing for the conjunction of hypotheses. The proposed classifier can be easily deployed for the purpose of conformal prediction and selective classification with reject and refine options by simply thresholding the consensus p-values. The theoretical analysis on the generalization error bound of the proposed classifier is provided and empirical studies on real data sets are conducted as well to demonstrate its effectiveness.

Conjunction Subspaces Test for Conformal and Selective Classification

TL;DR

A new classifier is presented, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision.

Abstract

In this paper, we present a new classifier, which integrates significance testing results over different random subspaces to yield consensus p-values for quantifying the uncertainty of classification decision. The null hypothesis is that the test sample has no association with the target class on a randomly chosen subspace, and hence the classification problem can be formulated as a problem of testing for the conjunction of hypotheses. The proposed classifier can be easily deployed for the purpose of conformal prediction and selective classification with reject and refine options by simply thresholding the consensus p-values. The theoretical analysis on the generalization error bound of the proposed classifier is provided and empirical studies on real data sets are conducted as well to demonstrate its effectiveness.

Paper Structure

This paper contains 32 sections, 4 theorems, 27 equations, 9 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Research Motivation
  3. Method Outline and Contributions
  4. Related Work
  5. Classification via Hypothesis Testing
  6. Conformal Prediction
  7. Selective Classification
  8. Random Projection Ensemble Classification
  9. Method
  10. Notations
  11. An Overview of COST
  12. Subspaces Selection
  13. P-value Calculation
  14. P-value Combination
  15. Selective Classification
  16. ...and 17 more sections

Key Result

Lemma 1

On the i-th subspace $(1\leq i\leq b_1)$, we can construct a threshold classifier as: $f(\hat{x},s_i)=c_{j} \ if\ \mathbb{I}(p_{i,j} \leq p_{(\hat{r}),\hat{c}})$. Then, these $b_1$ threshold classifiers can be aggregated to form a bagging predictor: The class label reported by $g(\hat{x})$ is equivalent to that of COST.

Figures (9)

  • Figure 1: An overview of COST. Firstly, a specified number of subspaces are chosen according to some criteria. Secondly, a p-value is calculated for each class on every subspace. Finally, the p-value combination method is used to merge p-values from all subspaces to obtain a consensus p-value for each class. Through the comparison among these k class p-values or the introduction of a significance level, we can fulfill the task of regular classification or selective classification.
  • Figure 2: Running time comparison of different classifiers. The running time was measured in seconds and all experiments were conducted on a PC equipped with an M1 CPU and 16GB of memory.
  • Figure 3: The comparison of COST, BCOPS, and GPS with respect to the refine rate.
  • Figure 4: The comparison of COST, BCOPS, and GPS with respect to the reject rate.
  • Figure 5: The effect of the value of $b_1$ on COST in terms of the JacAcc score.
  • ...and 4 more figures

Theorems & Definitions (12)

  • Lemma 1
  • proof
  • Definition 1
  • Lemma 2
  • proof
  • Definition 2
  • Lemma 3
  • proof
  • Definition 3
  • Definition 4
  • ...and 2 more