Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A frequentist local false discovery rate

Daniel Xiang, Jake A. Soloff, William Fithian

TL;DR

The paper introduces a frequentist counterpart to the local false discovery rate (lfdr) that avoids Bayesian priors by defining lfdr(t) as the relative frequency of null statistics at the point t. It proves that this lfdr provides calibrated forecasts and, when used with a threshold at $1/(1+\lambda)$, yields the optimal separable rejection under a weighted loss, while remaining estimable via standard density estimation methods. A novel finite-sample error criterion, the boundary FDR (bFDR), is proposed and controlled by the Support Line (SL) method under independence, linking local lfdr concepts to actionable multiple testing procedures. The approach is demonstrated across three applications—Gaussian graphical models, microbiome data, and aggregate nudges—highlighting calibration advantages and practical utility in settings where Bayesian modeling of dependence or exchangeability is questionable. Overall, the work provides a principled, prior-free framework for locally assessing discovery quality and for conducting multiple testing with finite-sample guarantees, complemented by connections to empirical Bayes techniques and nonparametric density estimation.

Abstract

The local false discovery rate (lfdr) of Efron et al. (2001) enjoys major conceptual and decision-theoretic advantages over the false discovery rate (FDR) as an error criterion in multiple testing, but is only well-defined in Bayesian models where the truth status of each null hypothesis is random. We define a frequentist counterpart to the lfdr based on the relative frequency of nulls at each point in the sample space. The frequentist lfdr is defined without reference to any prior, but preserves several important properties of the Bayesian lfdr: For continuous test statistics, $\text{lfdr}(t)$ gives the probability, conditional on observing some statistic equal to $t$, that the corresponding null hypothesis is true. Evaluating the lfdr at an individual test statistic also yields a calibrated forecast of whether its null hypothesis is true. Finally, thresholding the lfdr at $\frac{1}{1+λ}$ gives the best separable rejection rule under the weighted classification loss where Type I errors are $λ$ times as costly as Type II errors. The lfdr can be estimated efficiently using parametric or non-parametric methods, and a closely related error criterion can be provably controlled in finite samples under independence assumptions. Whereas the FDR measures the average quality of all discoveries in a given rejection region, our lfdr measures how the quality of discoveries varies across the rejection region, allowing for a more fine-grained analysis without requiring the introduction of a prior.

A frequentist local false discovery rate

TL;DR

The paper introduces a frequentist counterpart to the local false discovery rate (lfdr) that avoids Bayesian priors by defining lfdr(t) as the relative frequency of null statistics at the point t. It proves that this lfdr provides calibrated forecasts and, when used with a threshold at , yields the optimal separable rejection under a weighted loss, while remaining estimable via standard density estimation methods. A novel finite-sample error criterion, the boundary FDR (bFDR), is proposed and controlled by the Support Line (SL) method under independence, linking local lfdr concepts to actionable multiple testing procedures. The approach is demonstrated across three applications—Gaussian graphical models, microbiome data, and aggregate nudges—highlighting calibration advantages and practical utility in settings where Bayesian modeling of dependence or exchangeability is questionable. Overall, the work provides a principled, prior-free framework for locally assessing discovery quality and for conducting multiple testing with finite-sample guarantees, complemented by connections to empirical Bayes techniques and nonparametric density estimation.

Abstract

The local false discovery rate (lfdr) of Efron et al. (2001) enjoys major conceptual and decision-theoretic advantages over the false discovery rate (FDR) as an error criterion in multiple testing, but is only well-defined in Bayesian models where the truth status of each null hypothesis is random. We define a frequentist counterpart to the lfdr based on the relative frequency of nulls at each point in the sample space. The frequentist lfdr is defined without reference to any prior, but preserves several important properties of the Bayesian lfdr: For continuous test statistics, gives the probability, conditional on observing some statistic equal to , that the corresponding null hypothesis is true. Evaluating the lfdr at an individual test statistic also yields a calibrated forecast of whether its null hypothesis is true. Finally, thresholding the lfdr at gives the best separable rejection rule under the weighted classification loss where Type I errors are times as costly as Type II errors. The lfdr can be estimated efficiently using parametric or non-parametric methods, and a closely related error criterion can be provably controlled in finite samples under independence assumptions. Whereas the FDR measures the average quality of all discoveries in a given rejection region, our lfdr measures how the quality of discoveries varies across the rejection region, allowing for a more fine-grained analysis without requiring the introduction of a prior.

Paper Structure

This paper contains 24 sections, 20 theorems, 181 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Interpreting the frequentist lfdr
  3. Review of the Bayes two-groups model
  4. Three interpretations of the frequentist lfdr
  5. Limitations
  6. lfdr and FDR
  7. Estimating the lfdr
  8. Controlling the lfdr
  9. Comparison with FDR
  10. Controlling the boundary FDR
  11. Non-Uniform$(0,1)$ null distribution
  12. Applications
  13. Example 1: Gaussian graphical model
  14. Example 2: Microbiome data analysis
  15. Example 3: Aggregate analysis of nudges
  16. ...and 9 more sections

Key Result

Theorem 2.1

Suppose $z_1,\dots,z_m$ are jointly absolutely continuous. Then where $J$ is the (random) index of the statistic with $z_J=t$.

Figures (10)

  • Figure 1: Microbiome preservation example. The left panel shows a histogram of permutation $p$-values comparing relative abundance in fresh vs. eight-week-old preserved samples in each of $m=1147$ species, along with a nonparametric estimate of the null and alternative components of the mixture density using the empirical Bayes estimator of strimmer2008unified. The right panel shows the corresponding lfdr estimates, as well as the $q$-value of storey2002direct. While these lfdr estimates would be difficult to justify as posterior probabilities in a fully Bayesian analysis, they have natural interpretations in our frequentist framework.
  • Figure 2: Simulation example. The true means in \ref{['ex:gaussian-means']} are $\mu_1=\dots=\mu_{m_1}=2$ and $\mu_i=0$ for $i = m_1+1,\dots,m$, where $m=3000$ and $m_1=150$, so the true null proportion is $95\%$. The calibration curve for the frequentist lfdr is plotted in yellow. We repeat the experiment $10^4$ times to assess calibration.
  • Figure 3: The order statistics of $m=500$$p$-values are plotted against their rank. They are generated with $\bar{\pi}_0=0.5$ where null $p$-values (red) are i.i.d. Uniform$(0,1)$ and alternative $p$-values (blue) are i.i.d. Beta$(0.05,1)$. The bFDR is approximated by the fraction of nulls among the largest 15 $p$-values below $0.1$.
  • Figure 4: Histogram of $t$-statistics in the GGM example, with dimensions: $d=80$, $n=10d$, $m=3,160$. The null distribution is $t_{n-d}$.
  • Figure 5: Shown above is the histogram of one-sided $p$-values falling below $0.025$, adjusted for selection by multiplying by 40 (the reciprocal of the 2.5% one-sided significance threshold). The $\textnormal{BH}(q)$ threshold for $q=10\%$ is around $0.27$ (or $\approx 0.007$ on the scale of the unadjusted $p$-values), below which there are 202 rejections. The estimated FDP near the edge of the rejection set (red) is around 32%.
  • ...and 5 more figures

Theorems & Definitions (43)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem 2.3
  • Theorem 3.1
  • Theorem 3.2
  • Remark 3.1
  • Proposition 4.1
  • Theorem 5.1
  • proof : Proof of Theorem \ref{['main-RFDR-control']}.
  • Theorem 5.2
  • ...and 33 more