On min-Storey estimators for multiple testing and conformal novelty detection

Abstract

In a multiple testing task, finding an appropriate estimator of the proportion $π_0$ of non-signal in the data to boost power of false discovery rate (FDR) controlling procedures is a long-standing research theme, sometimes referred to as 'adaptive FDR control'. The interest in this theme has been reinforced in the recent years with conformal novelty detection, for which it turns out that similar tools can be used in combination with any 'blackbox' machine learning algorithm. Nevertheless, perhaps surprisingly, finding a solution for 'adaptive FDR control' that is optimal in a broad sense is still an open problem. This paper fills this gap by introducing new $π_0$-estimators, referred to as min-Storey (MS) and interval-min-Storey (IMS), which are built upon the so-called 'Storey estimator'. Plugging these estimators in the adaptive Benjamini-Hochberg (BH) procedure is shown to deliver FDR control both in the independent and conformal settings. In addition, these methods satisfy an optimal power property over any (regular) alternative distribution. The excellent behaviors of the new adaptive procedures are illustrated with numerical experiments both in the independent and conformal models for various distribution structures.

Figures (23)

• Figure 1: Comparison of null proportion estimators for conformal $p$-values. The left panel shows the empirical distribution of the conformal $p$-values. The test and calibration samples both have size $1000$. Null and calibration scores are drawn independently from $U(0,1)$; for the alternatives, we generate independent scores $S_i = \Phi(X_i)$ with $X_i \sim 0.5\mathcal{N}(1,9)+0.5\mathcal{N}(0.2,0.09)$. The choice of the alternative mixture produces a mid-range bump (weak signals from the component $\mathcal{N}(0.2,0.09)$) in the $p$-value histogram and an excess of large $p$-values (the high-variance component $\mathcal{N}(1,9)$ yields significantly negative $X_i$ and thus considerably small $S_i$). The true null proportion $\pi_0 = 0.5$. The right panel shows various null proportion estimators $\hat{\pi}_0$ and the resulting number of rejections $R$ from the associated adaptive BH ($400$ repeats; standard errors in parentheses). Our proposal MS and IMS produce the smallest $\pi_0$-estimation and the largest rejection number in average.
• Figure 2: Visualization of IMS with $\hat{\kappa}$. The left panel shows the empirical distribution of the $p$-values together with $\hat{\pi}_{0,\epsilon,\underline{\pi}_0,\kappa}^{\mathrm{IMS}}$ nondecreasing in $\kappa$. The middle panel plots the empirical CDF against the $p$-value level or $\kappa$. It shows that $\hat{\kappa}$ is the largest value for which the empirical CDF at $\kappa$ exceeds $\kappa\hat{\pi}_{0,\epsilon,\underline{\pi}_0,\kappa}^{\mathrm{IMS}}/\alpha$. The red dot marks the rejection threshold $\hat{\kappa}$, which is larger than its blue counterpart under the fixed choice $\kappa=0.5$. The right panel is obtained by flipping the x and y axes of the middle panel, thereby plotting the ordered $p$-values against the normalized rank $q$, i.e., the rank of p-value divided by $m$. The red dashed curve denotes $\alpha q / \hat{\pi}_{0,\epsilon,\underline{\pi}_0,\kappa}^{\mathrm{IMS}}$ with $q = \hat{F}_m(\kappa)$, while the blue dashed curve corresponds to $\alpha q / \hat{\pi}_{0,\epsilon,\underline{\pi}_0,0.5}^{\mathrm{IMS}}$.
• Figure 6: Numerical evaluation of $c(s,\epsilon)$
• Figure 7: Numerical evaluation of $d(s,\epsilon)$
• Figure 10: Comparison of null proportion estimators with independent$p$-values under the many signal setting (a.3) in Section \ref{['sec:simulation']}. For MS and IMS, we use a conservative $\underline{\pi}_0 = 0.5 > \pi_0 =0.2$.

Theorems & Definitions (46)

• Theorem 3.1
• Corollary 3.2
• Corollary 3.3
• Remark 3.1
• Definition 1
• Theorem 4.1
• Definition 2
• Theorem 4.2
• Definition 3
• Theorem 4.3