Conformal Inference for Open-Set and Imbalanced Classification

TL;DR

A conformal prediction method for classification in highly imbalanced and open-set settings, where there are many possible classes and not all may be represented in the data, is presented and leads to prediction sets with valid coverage even in challenging open-set scenarios with infinite numbers of possible labels.

Abstract

This paper presents a conformal prediction method for classification in highly imbalanced and open-set settings, where there are many possible classes and not all may be represented in the data. Existing approaches require a finite, known label space and typically involve random sample splitting, which works well when there is a sufficient number of observations from each class. Consequently, they have two limitations: (i) they fail to provide adequate coverage when encountering new labels at test time, and (ii) they may become overly conservative when predicting previously seen labels. To obtain valid prediction sets in the presence of unseen labels, we compute and integrate into our predictions a new family of conformal p-values that can test whether a new data point belongs to a previously unseen class. We study these p-values theoretically, establishing their optimality, and uncover an intriguing connection with the classical Good--Turing estimator for the probability of observing a new species. To make more efficient use of imbalanced data, we also develop a selective sample splitting algorithm that partitions training and calibration data based on label frequency, leading to more informative predictions. Despite breaking exchangeability, this allows maintaining finite-sample guarantees through suitable re-weighting. With both simulated and real data, we demonstrate our method leads to prediction sets with valid coverage even in challenging open-set scenarios with infinite numbers of possible labels, and produces more informative predictions under extreme class imbalance.

Figures (15)

• Figure 1: Performance of conformal prediction sets constructed using different methods on synthetic data generated from a Dirichlet process model, as a function of the concentration parameter $\theta$. Larger $\theta$ values correspond to a higher probability of unseen labels at test time. The nominal coverage level is $90\%$. The conformal Good--Turing classification (CGTC) method maintains valid marginal coverage even when new labels are common by incorporating a "catch-all" joker symbol into the prediction sets. When combined with the selective sample-splitting strategy, it also produces smaller and more informative prediction sets. Error bars represent $\pm 1.96$ standard errors.
• Figure 2: Coverage of conformal prediction sets constructed using different methods on synthetic data from a Dirichlet process model, as in Figure \ref{['fig:dp-main-four-panel']}, stratified by test-label frequency. The standard approach fails to achieve coverage for very rare labels, whereas the conformal Good--Turing method is valid for both rare and common labels.
• Figure 3: Percentage of conformal Good--Turing prediction sets including the joker symbol under different $p$-value constructions for testing the null hypothesis $H_{\text{unseen}}$, which states that the test point corresponds to a new label, on synthetic data. The feature-based method (XGT)—corresponding to the results shown in Figure \ref{['fig:dp-main-four-panel']}—makes the most parsimonious use of the joker by attaining the highest power to reject $H_{\text{unseen}}$. In contrast, the deterministic feature-blind approach (GT) is overly conservative, and its randomized variant (RGT) reduces this conservatism only slightly.
• Figure 4: Performance of conformal prediction sets constructed using different methods on face recognition data from the CelebA resource, as a function of the total number of labeled data points. Conformal Good--Turing classification (CGTC) maintains valid marginal coverage across all settings and achieves smaller prediction sets when combined with the selective sample splitting strategy. Error bars indicate $1.96$ standard errors.
• Figure A1: Allocation of significance levels $(\alpha_{\text{class}}, \alpha_{\text{unseen}}, \alpha_{\text{seen}})$ when applying the proposed conformal Good--Turing classification method with a total budget $\alpha = 0.1$ on synthetic data generated from a Dirichlet process model, as in Figure \ref{['fig:dp-main-four-panel']}. Error bars indicate $1.96$ standard errors.

Theorems & Definitions (24)

• Example 1: Dirichlet Process Model
• Theorem 1
• Theorem 2
• Proposition 1
• Theorem 3
• Theorem 4
• Theorem 5
• Proposition 2
• Theorem 6
• Lemma A1