Testing for Outliers with Conformal p-values

TL;DR

This paper proposes a solution based on conformal inference, a broadly applicable framework which yields p- values that are marginally valid but mutually dependent for different test points, and proves these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense.

Abstract

This paper studies the construction of p-values for nonparametric outlier detection, taking a multiple-testing perspective. The goal is to test whether new independent samples belong to the same distribution as a reference data set or are outliers. We propose a solution based on conformal inference, a broadly applicable framework which yields p-values that are marginally valid but mutually dependent for different test points. We prove these p-values are positively dependent and enable exact false discovery rate control, although in a relatively weak marginal sense. We then introduce a new method to compute p-values that are both valid conditionally on the training data and independent of each other for different test points; this paves the way to stronger type-I error guarantees. Our results depart from classical conformal inference as we leverage concentration inequalities rather than combinatorial arguments to establish our finite-sample guarantees. Furthermore, our techniques also yield a uniform confidence bound for the false positive rate of any outlier detection algorithm, as a function of the threshold applied to its raw statistics. Finally, the relevance of our results is demonstrated by numerical experiments on real and simulated data.

Figures (22)

• Figure 1: Visualization of the joint distribution of the conformal p-values. The distribution of $\hat{s}(x)$ is the same for calibration and inlier test points. The conformal p-value for each test point is the number of calibration points to its left, divided by the total number of calibration points plus one, as in \ref{['eq:marginal-pvals-def']}.
• Figure 2: Distribution of the false positive rate obtained by thresholding marginal conformal p-values at levels $\alpha=0.01$ and $\alpha=0.1$, as a function of the number of calibration points.
• Figure 3: Illustration of Theorem \ref{['thm:generic']} with $n=1000$ and $\delta=0.1$. The orange and yellow curves give the sequences derived by the generalized Simes inequality with $k = 500$ and the DKWM inequality, respectively. The blue and green curves (very close to each other) give the corresponding sequences obtained with the asymptotic and Monte Carlo adjustments described below. The right panel zooms in on small indices.
• Figure 4: Comparison of different adjustment functions, with $n=1000$ and $\delta=0.1$. In the zoomed-in panel on the right-hand-side, the Simes (orange) and Monte Carlo (green) curves cannot be distinguished.
• Figure 5: Power analysis of different adjustments for marginal conformal p-values under 3 alternative settings. The effective level resulting from the p-value adjustment for a test at nominal level $\alpha=0.05$ (dashed horizontal line) is plotted as a function of the number of calibration samples, assuming the number of test points $m$ grows as $\sqrt{n}$. (a) Testing a single hypothesis. (b) FWER control with a single strong signal (here the values for DKWM are all equal to 0). (c) Testing a global null with Fisher's combination test.

Theorems & Definitions (72)

• Lemma 1
• Theorem 1: Type-I error of Fisher's combination test
• Definition 1
• Theorem 2: Conformal p-values are PRDS
• Corollary 1: Benjamini and Yekutieli benjamini2001control
• Remark 1
• Theorem 3: Storey's BH with conformal p-values controls the FDR
• Proposition 1: Pointwise FPR of marginal conformal p-values, from vovk2012conditional
• Theorem 4: Conditional p-value adjustment
• Proposition 2: Generalized Simes Inequality, from Equation (3.5) in sarkar2008generalizing