Synthetic-Powered Multiple Testing with FDR Control

TL;DR

SynthBH is introduced, a synthetic-powered multiple testing procedure that safely leverages auxiliary or synthetic data that enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality.

Abstract

Multiple hypothesis testing with false discovery rate (FDR) control is a fundamental problem in statistical inference, with broad applications in genomics, drug screening, and outlier detection. In many such settings, researchers may have access not only to real experimental observations but also to auxiliary or synthetic data -- from past, related experiments or generated by generative models -- that can provide additional evidence about the hypotheses of interest. We introduce SynthBH, a synthetic-powered multiple testing procedure that safely leverages such synthetic data. We prove that SynthBH guarantees finite-sample, distribution-free FDR control under a mild PRDS-type positive dependence condition, without requiring the pooled-data p-values to be valid under the null. The proposed method adapts to the (unknown) quality of the synthetic data: it enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality. We demonstrate the empirical performance of SynthBH on tabular outlier detection benchmarks and on genomic analyses of drug-cancer sensitivity associations, and further study its properties through controlled experiments on simulated data.

Figures (10)

• Figure 1: Comparison of conformal outlier detection methods on three tabular datasets. The target FDR level is $\alpha=10\%$ and $\varepsilon=10\%$. Top: detection rate (empirical power). Bottom: false discovery proportion.
• Figure 2: Power as a function of the empirical FDR for BH (real) and our proposed method SynthBH across different outlier detection datasets: (a) Shuttle, (b) Credit-card, and (c) KDDCup99. The first row corresponds to a trimming proportion of 2%, and the second row corresponds to a trimming proportion of 0%, representing a scenario where synthetic data are not useful. Here, $\alpha\in [0,20\%]$.
• Figure 3: Comparison of multiple testing methods on GDSC genomic data for 100 tissue-feature-drug hypotheses. The target FDR level is $\alpha=10\%$ and $\varepsilon=10\%$. Left: number of rejections. Right: average ground-truth score of the rejection set.
• Figure S1: Comparison of multiple hypothesis testing as a function of the real data sample size $n$. Left: average detection rate (power). Right: average false discovery proportion.
• Figure S2: Comparison of multiple hypothesis testing as a function of the synthetic signal strength $\rho_{\text{synth}}$. Results are shown for (a) high-quality synthetic data and (b) worst-case synthetic data (in terms of error). Left: average detection rate (power). Right: average false discovery proportion.

Theorems & Definitions (13)

• Definition 4.1: benjamini2001control
• Definition 4.2
• Definition 4.3
• Theorem 4.4: SynthBH controls the FDR
• Lemma 5.1
• Theorem 5.2: Outlier detection via SynthBH with FDR control
• Remark 5.3
• Lemma B.1: Static reduction of SynthBH
• proof
• Lemma C.1