Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the universal calibration of heavy-tailed combination tests

Parijat Chakraborty, F. Richard Guo, Kerby Shedden, Stilian Stoev

Abstract

It is often of interest to test a global null hypothesis using multiple, possibly dependent $p$-values by combining their strengths while controlling the type-I error. Recently, several heavy-tailed combination tests, such as the harmonic mean test and the Cauchy combination test, have been proposed: they transform $p$-values into heavy-tailed random variables before combining them into a single test statistic. The resulting tests, which are calibrated under some form of independence assumption among the $p$-values, have been shown to be rather robust to dependence asymptotically as the $α$ level gets small. Yet, it has remained an open problem to understand this general phenomenon and characterize how such tests behave under dependence. Using the framework of multivariate regular variation from extreme value theory, we show that for a class of combination tests that are homogeneous, the asymptotic level of the test can be expressed using the angular measure under multivariate regular variation. This measure characterizes the dependence of the transformed heavy-tailed variables in their upper tails, or equivalently, the dependence of the $p$-values near zero. We use this result to study several tests. The harmonic mean test, which coincides with the Pareto linear combination test, is shown to be universally calibrated regardless of the tail dependence; further, this test is shown to be the only one that achieves universal calibration among all homogeneous heavy-tailed combination tests. In contrast, the Cauchy combination test is shown to be universally honest but often conservative; the Dunn-Šidák correction, also known as the Tippett's method, while being honest, is calibrated if and only if the underlying $p$-values are independent near zero. These theoretical findings are corroborated with simulations and an application to independence testing with survey data.

On the universal calibration of heavy-tailed combination tests

Abstract

It is often of interest to test a global null hypothesis using multiple, possibly dependent -values by combining their strengths while controlling the type-I error. Recently, several heavy-tailed combination tests, such as the harmonic mean test and the Cauchy combination test, have been proposed: they transform -values into heavy-tailed random variables before combining them into a single test statistic. The resulting tests, which are calibrated under some form of independence assumption among the -values, have been shown to be rather robust to dependence asymptotically as the level gets small. Yet, it has remained an open problem to understand this general phenomenon and characterize how such tests behave under dependence. Using the framework of multivariate regular variation from extreme value theory, we show that for a class of combination tests that are homogeneous, the asymptotic level of the test can be expressed using the angular measure under multivariate regular variation. This measure characterizes the dependence of the transformed heavy-tailed variables in their upper tails, or equivalently, the dependence of the -values near zero. We use this result to study several tests. The harmonic mean test, which coincides with the Pareto linear combination test, is shown to be universally calibrated regardless of the tail dependence; further, this test is shown to be the only one that achieves universal calibration among all homogeneous heavy-tailed combination tests. In contrast, the Cauchy combination test is shown to be universally honest but often conservative; the Dunn-Šidák correction, also known as the Tippett's method, while being honest, is calibrated if and only if the underlying -values are independent near zero. These theoretical findings are corroborated with simulations and an application to independence testing with survey data.

Paper Structure

This paper contains 26 sections, 25 theorems, 172 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Multivariate regular variation and asymptotic calibration of combination tests
  3. Multivariate regular variation
  4. Examples of multivariate regular variation
  5. A general approach to calibrating heavy-tailed combination tests
  6. Universal calibration and honesty
  7. Characterizing universal calibration
  8. On integrals under linear constraints
  9. Characterization
  10. Tippett's method, Dunn--Šidák correction and Fréchet combination test
  11. Application to multiple data splitting
  12. Simulation studies
  13. Calibration
  14. Power
  15. An application to independence testing of multidimensional physiological traits
  16. ...and 11 more sections

Key Result

Theorem 1

Let $X = (X_i)_{i=1}^d$ be a random vector in $\mathbb R^d$.

Figures (5)

  • Figure 1: Type-I error relative to the nominal level of combination tests under a 10-dimensional multivariate $t$-copula with $\nu$ degrees of freedom and an autoregressive shape matrix in \ref{['eqs:t-sigma']}. The curves of Pareto and Cauchy+ almost overlap. The results are computed from $10^6$ replications and the standard errors are negligible.
  • Figure 2: Pairwise plots of the combined $p$-values in the multivariate $t$ simulation setting with $\nu=3$, $d=10$ and $\rho=0.1$. Left: $\tau=0$; Right: $\tau = 4$.
  • Figure 3: Power of combination tests under $\alpha=0.05$ for testing $\mu=0$ relative to the oracle likelihood ratio test. Each combination test is computed from $d$ two-sided $p$-values corresponding to the coordinates of $t_{\nu}(\tau \eta, \Sigma)$, where $\Sigma$ is of autoregressive type with $\rho=0.1$. The curves of Pareto and Cauchy+ almost overlap. The results are computed from $10^6$ replications and the standard errors are negligible.
  • Figure S1: Type-I error relative to the nominal level of combination tests under a 10-dimensional multivariate $t$-copula with $\nu$ degrees of freedom and an exchangeable shape matrix $\Sigma = (\rho^{\mathbb{I}_{i\neq j}})_{d\times d}$. The curves of Pareto and Cauchy+ almost overlap. The results are computed from $10^6$ replications.
  • Figure S2: Power of combination tests for testing $\mu=0$ relative to the oracle likelihood ratio test. Each combination test is computed from $d$ two-sided $p$-values corresponding to the coordinates of $t_{\nu}(\tau \eta, \Sigma)$, where $\Sigma = (\rho^{\mathbb{I}_{i\neq j}})_{d\times d}$ with $\rho=0.1$. The curves of Pareto and Cauchy+ almost overlap. The results are computed from $10^6$ replications.

Theorems & Definitions (64)

  • Definition 1: asymptotic calibration and honesty
  • Definition 2: upper tail dependence coefficient and asymptotic independence
  • Definition 3
  • Remark 1
  • Theorem 1: Tail index theorem
  • Theorem 2
  • Lemma 1: see Proposition 2.5 in jansen:neblung:stoev:2023
  • Lemma 2: Generalized Breiman's lemma
  • Example 1: multivariate $t$-distribution
  • Example 2: heavy-tailed factor models
  • ...and 54 more