Comparing three learn-then-test paradigms in a multivariate normal means problem

TL;DR

This work analyzes three learn-then-test paradigms for multiple testing in a multivariate normal means framework: information splitting (Split BH), null augmentation (BONuS), and post-learning adjustment (In-sample BH). It derives unified asymptotic power expressions under two priors (point-mass and one-dimensional subspace) and shows that, in the limit, post-learning adjustment is most powerful, followed by null augmentation, then information splitting; importantly, BONuS can approach In-sample BH power when the augmentation fraction vanishes. A key finding is that the optimal BONuS augmentation scales as $\tilde{m}=\Theta(\sqrt{m})$, challenging the prior heuristics, while Split BH’s optimal split proportion lies strictly between 0 and 1 and depends on the learning difficulty via the alignment of the learned direction. The results provide a theoretical foundation for choosing LTT paradigms and tuning masking parameters, with implications for the design of powerful, scalable multiple-testing procedures in high dimensions.

Abstract

Many modern procedures use the data to learn a structure and then leverage it to test many hypotheses. If the entire data is used at both stages, analytical or computational corrections for selection bias are required to ensure validity (post-learning adjustment). Alternatively, one can learn and/or test on masked versions of the data to avoid selection bias, either via information splitting or null augmentation}. Choosing among these three learn-then-test paradigms, and how much masking to employ for the latter two, are critical decisions impacting power that currently lack theoretical guidance. In a multivariate normal means model, we derive asymptotic power formulas for prototypical methods from each paradigm -- variants of sample splitting, conformal-style null augmentation, and resampling-based post-learning adjustment -- quantifying the power losses incurred by masking at each stage. For these paradigm representatives, we find that post-learning adjustment is most powerful, followed by null augmentation, and then information splitting. Moreover, null augmentation can be nearly as powerful as post-learning adjustment, while avoiding its challenges: the power of the former approaches that of the latter if the number of nulls used for augmentation is a vanishing fraction of the number of hypotheses. We also prove for a tractable proxy that the optimal number of nulls scales as the square root of the number of hypotheses, challenging existing heuristics. Finally, we characterize optimal tuning for information splitting by identifying an optimal split fraction and tying it to the difficulty of the learning problem. These results establish a theoretical foundation for key decisions in the deployment of learn-then-test methods.

Figures (20)

• Figure 1: Representative LTT methods in the empirical Bayes multivariate normal means testing problem. The colored squares represent numeric values.
• Figure 2: A framework for dissecting the asymptotic power of the three LTT methods under two prior choices, mirroring the structures of the methods themselves (Figure \ref{['fig:common-paradigm']}).
• Figure 3: Asymptotic alignment of learned direction $\hat{\bm v}$ with the true direction $\bm v$.
• Figure 4: The powers of Split BH (top left) and BONuS (top right) versus their tuning parameters, and compared to that of In-sample BH, for the point prior. The powers of the procedures are compared as a function of the limiting alignment of the learned direction with the true direction (bottom left). We also compare the empirical maximum powers attained by Split BH and BONuS, with the power of In-sample BH, which has no tuning parameters, as a function of the signal strength $h$ (bottom right).
• Figure 5: Optimal Split BH learning proportion versus ease of learning, $\tau_{\text{mean}}$. Point marks default setting $(\gamma, c, h) = (0.2, 0.2, 4)$.

Theorems & Definitions (86)

• definition 1: BH TPR function
• lemma 1: Quality of mean learner on original data
• proposition 1: Quality of mean learner on masked data
• proposition 2: Asymptotic distributions of scores
• theorem 1: Asymptotic power under point mass prior
• remark 1: Power of BONuS
• remark 2: Power comparison
• lemma 2: Quality of PCA learner on original data
• proposition 3: Quality of PCA learner on masked data
• proposition 4: Asymptotic distributions of scores