Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tiny but uniform improvements of adaptive BH procedures via compound e-values

Nikolaos Ignatiadis, Ruodu Wang, Aaditya Ramdas

Abstract

After the seminal Benjamini-Hochberg (BH) procedure for controlling the false discovery rate (FDR) was proposed, dozens of papers have attempted to improve its power by adapting to the unknown proportion of nulls. We observe that most null proportion estimates are simply compound e-values in disguise, and thus most adaptive FDR procedures can be interpreted as instances of the e-weighted BH (ep-BH) procedure of Ignatiadis, Wang, and Ramdas [2024], i.e., the BH procedure weighted by compound e-values. This lens helps us show that most existing procedures are inadmissible, and we provide uniform improvements to them. While the improvements are small in practice, they still come for free (without additional assumptions), and help unify the literature. We also use our "leave-one-out ep-BH method" to design a new method with finite-sample FDR control for the simultaneous t-test setting.

Tiny but uniform improvements of adaptive BH procedures via compound e-values

Abstract

After the seminal Benjamini-Hochberg (BH) procedure for controlling the false discovery rate (FDR) was proposed, dozens of papers have attempted to improve its power by adapting to the unknown proportion of nulls. We observe that most null proportion estimates are simply compound e-values in disguise, and thus most adaptive FDR procedures can be interpreted as instances of the e-weighted BH (ep-BH) procedure of Ignatiadis, Wang, and Ramdas [2024], i.e., the BH procedure weighted by compound e-values. This lens helps us show that most existing procedures are inadmissible, and we provide uniform improvements to them. While the improvements are small in practice, they still come for free (without additional assumptions), and help unify the literature. We also use our "leave-one-out ep-BH method" to design a new method with finite-sample FDR control for the simultaneous t-test setting.
Paper Structure (26 sections, 12 theorems, 65 equations, 4 figures, 1 table, 4 algorithms)

This paper contains 26 sections, 12 theorems, 65 equations, 4 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background on compound e-values and ep-BH
  3. New results on FDR control with ep-BH
  4. Adaptive multiple testing with ep-BH
  5. Homogeneous null-proportion adaptivity
  6. Heterogeneous weighted null-proportion adaptivity
  7. Combining null proportion estimators
  8. Simultaneous t-tests and numerical results
  9. Acknowledgments.
  10. The ep-BH procedure and tau-censoring
  11. Omitted proofs and details for null proportion estimators
  12. Proof of Theorem \ref{['theorem:epbh_fdr']}
  13. Proof of Theorem \ref{['theorem:loo_epbh']}
  14. Proof of Theorem \ref{['theorem:loo_epbh_censoring']}
  15. Proof of Theorem \ref{['proposition:generic_loo_adaptive']}
  16. ...and 11 more sections

Key Result

Proposition 2.3

Let $E_1,\ldots,E_K$ be compound e-variables and $P_1,\ldots,P_K$ be p-variables. Assume that for each $k \in \mathcal{N}$, $P_k$ is independent of $E_k$. Then $Q_k := P_k/E_k$ are compound p-variables.

Figures (4)

  • Figure 1: Rejection regions for two hypotheses ($K=2$). Colors and hatching indicate 0/1/2 discoveries. Our proposed Storey+ dominates standard Storey tuned with same parameters $\alpha$ and $\tau$. Fig. \ref{['fig:regions_alpha04_tau04']} shows an analogous plot for $\alpha=\tau=0.4$.
  • Figure S1: Power (a) and FDR (b) versus effect size $\xi$ for 7 different methods for multiple testing with simultaneous t-tests. We use the weighting function $\psi(u)=u^4$. The different facets correspond to different sample size ($n \in \{2,5,20\}$) and number of alternative hypotheses $K_1 \in \{2,5,100\}$, where the total number of hypotheses is $K=200$. In this simulation study, W-LOO-Storey zhao2024censored and W-LOO-Storey+ (ours) performed identically, which is not surprising given the tiny improvement that we expect the latter to provide over the former. For this reason, the legend only shows W-LOO-Storey+. All methods control FDR at the nominal $\alpha=0.1$. LOO-Var+ (ours) and W-LOO-Storey+ have the most power across facets.
  • Figure S2: Analogous to Figure \ref{['fig:psi1']} with a different choice of weighting function, namely $\psi(u)=u^2 \mathds{1}_{\{u \geq 1 \}}$.
  • Figure S3: Rejection regions for two hypotheses ($K=2$). Colors and hatching indicate 0/1/2 discoveries. Our proposed Storey+ dominates standard Storey tuned with same parameters $\alpha$ and $\tau$. This figure is analogous to Fig. \ref{['fig:regions_alpha04_tau02']} in the main text but with different choices of $\alpha$ and $\tau$.

Theorems & Definitions (38)

  • Definition 2.1: Compound p-variables and e-variables
  • Definition 2.2: The ep-BH procedure
  • Proposition 2.3
  • Definition 3.1: P-independence
  • Definition 3.2: Positive regression dependence on a subset
  • Theorem 3.3
  • Theorem 3.4
  • Example 4.1: Storey
  • Remark 4.2: $\tau$-censoring
  • Example 4.3: Modified Pounds-Cheng, MPC
  • ...and 28 more