Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using prior information to boost power in correlation structure support recovery

Ziyang Ding, David Dunson

TL;DR

The paper tackles high-dimensional correlation structure testing by boosting power via a frequentist assisted by Bayes (FAB) framework that leverages prior information. It adapts FAB to correlation testing through Fisher-transformed statistics, introducing a linking model that borrows information from external data or within-dataset tests, and employs a divide-and-conquer strategy for scalability. The authors provide theoretical justification for asymptotic Type I error control, and demonstrate power gains in simulations and real genomics data (Cancer Dependency Map) while comparing to UMPU and AdaPT. The approach offers a flexible, error-safe method for more effective correlation support recovery with practical implications in biology and finance, particularly when informative external or internal priors are available.

Abstract

Hypothesis testing of structure in correlation and covariance matrices is of broad interest in many application areas. In high dimensions and/or small to moderate sample sizes, high error rates in testing is a substantial concern. This article focuses on increasing power through a frequentist assisted by Bayes (FAB) procedure. This FAB approach boosts power by including prior information on the correlation parameters. In particular, we suppose there is one of two sources of prior information: (i) a prior dataset that is distinct from the current data but related enough that it may contain valuable information about the correlation structure in the current data; and (ii) knowledge about a tendency for the correlations in different parameters to be similar so that it is appropriate to consider a hierarchical model. When the prior information is relevant, the proposed FAB approach can have significant gains in power. A divide-and-conquer algorithm is developed to reduce computational complexity in massive testing dimensions. We show improvements in power for detecting correlated gene pairs in genomic studies while maintaining control of Type I error or false discover rate (FDR).

Using prior information to boost power in correlation structure support recovery

TL;DR

The paper tackles high-dimensional correlation structure testing by boosting power via a frequentist assisted by Bayes (FAB) framework that leverages prior information. It adapts FAB to correlation testing through Fisher-transformed statistics, introducing a linking model that borrows information from external data or within-dataset tests, and employs a divide-and-conquer strategy for scalability. The authors provide theoretical justification for asymptotic Type I error control, and demonstrate power gains in simulations and real genomics data (Cancer Dependency Map) while comparing to UMPU and AdaPT. The approach offers a flexible, error-safe method for more effective correlation support recovery with practical implications in biology and finance, particularly when informative external or internal priors are available.

Abstract

Hypothesis testing of structure in correlation and covariance matrices is of broad interest in many application areas. In high dimensions and/or small to moderate sample sizes, high error rates in testing is a substantial concern. This article focuses on increasing power through a frequentist assisted by Bayes (FAB) procedure. This FAB approach boosts power by including prior information on the correlation parameters. In particular, we suppose there is one of two sources of prior information: (i) a prior dataset that is distinct from the current data but related enough that it may contain valuable information about the correlation structure in the current data; and (ii) knowledge about a tendency for the correlations in different parameters to be similar so that it is appropriate to consider a hierarchical model. When the prior information is relevant, the proposed FAB approach can have significant gains in power. A divide-and-conquer algorithm is developed to reduce computational complexity in massive testing dimensions. We show improvements in power for detecting correlated gene pairs in genomic studies while maintaining control of Type I error or false discover rate (FDR).
Paper Structure (15 sections, 3 theorems, 22 equations, 10 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 3 theorems, 22 equations, 10 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Fisher-transformed correlation coefficient
  4. FAB Testing
  5. Methodology
  6. FAB Correlation Testing
  7. Lg: Sampling Model Using External Data
  8. Lg: Sampling Model Using Internal Data
  9. FAB Correlation Structure Testing
  10. Simulation
  11. Lg: Sampling Model Using External Data
  12. Lg: Sampling Model Using Internal Data
  13. Application
  14. Discussion
  15. Acknowledgement

Key Result

Theorem 2.1

hoff2021smaller Let $Z$ and $b$ be independent random variables with $Z \sim N(0,1)$. Then $\operatorname{Pr}(1- |\Phi(Z+b)-\Phi(-Z)|<u)=u$

Figures (10)

  • Figure 1: Divide and Conquer Group Assignment
  • Figure 2: UMPU vs FAB $p$-values when $\widehat{\bm{z}}_j^{\prime} = \widehat{\bm{z}}_{-j}^{\text{ext}}$. Figure \ref{['fig: Idpt UMPU FAB power line']} indicates ranked UMPU and FAB $p$-values. FAB has smaller $p$-values. Figure \ref{['fig: Idpt UMPU FAB scatter']} plots UMPU versus FAB $p$-values.
  • Figure 3: UMPU and FAB $p$-values distribution, when $\widehat{\bm{z}}_j^{\prime} = \widehat{\bm{z}}_{-j}^{\text{ext}}$. Under null (\ref{['fig: Idpt UMPU FAB NULL']}), UMPU and FAB $p$-values are uniform. Under alternative (\ref{['fig: Idpt UMPU FAB ALTER']}) FAB leads to more small $p$-values.
  • Figure 4: UMPU vs FAB $p$-values when $\widehat{\bm{z}}_j^{\prime} =\widehat{\bm{z}}_{-j}$. Figure \ref{['fig: Idpt UMPU FAB power line']} indicates ranked UMPU and FAB $p$-values. FAB has smaller $p$-values. Figure \ref{['fig: Idpt UMPU FAB scatter']} indicates UMPU versus FAB $p$-values
  • Figure 5: UMPU and FAB $p$-values distribution, when $\widehat{\bm{z}}_j^{\prime} =\widehat{\bm{z}}_{-j}$. Under null (\ref{['fig: boot UMPU FAB NULL']}), UMPU and FAB $p$-values are uniform. Under alternative (\ref{['fig: boot UMPU FAB ALTER']}) FAB leads to more small $p$-values.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Theorem 2.1
  • Theorem 3.1
  • Theorem 3.2