Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Limitations of Randomization Tests in Finite Samples

Deniz Dutz, Xinyi Zhang

Abstract

Randomization tests deliver exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. In practice, achieving these guarantees often requires stronger conditions than the null hypothesis of primary interest. For example, sign-change tests of mean zero require symmetry and need not control finite-sample size for non-symmetric mean-zero distributions. We investigate whether the mismatch between the null and the invariance conditions required for exactness reflects the use of particular transformations or a more fundamental limitation. We provide a simple necessary and sufficient condition for a null hypothesis to admit a randomization test. Applying this framework to one-sample problems, we characterize the nulls that admit randomization tests on finite supports and derive impossibility results on continuous supports. In particular, we show that several common nulls, including mean zero, do not admit randomization tests. We further show that, among one-sample tests using linear group actions, the admissible nulls are limited to subsets of symmetric or Gaussian distributions. These results confirm that the absence of exact finite-sample validity is inherent for many commonly studied nulls and that practitioners using existing tests are not foregoing feasible exact alternatives.

Limitations of Randomization Tests in Finite Samples

Abstract

Randomization tests deliver exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. In practice, achieving these guarantees often requires stronger conditions than the null hypothesis of primary interest. For example, sign-change tests of mean zero require symmetry and need not control finite-sample size for non-symmetric mean-zero distributions. We investigate whether the mismatch between the null and the invariance conditions required for exactness reflects the use of particular transformations or a more fundamental limitation. We provide a simple necessary and sufficient condition for a null hypothesis to admit a randomization test. Applying this framework to one-sample problems, we characterize the nulls that admit randomization tests on finite supports and derive impossibility results on continuous supports. In particular, we show that several common nulls, including mean zero, do not admit randomization tests. We further show that, among one-sample tests using linear group actions, the admissible nulls are limited to subsets of symmetric or Gaussian distributions. These results confirm that the absence of exact finite-sample validity is inherent for many commonly studied nulls and that practitioners using existing tests are not foregoing feasible exact alternatives.

Paper Structure

This paper contains 17 sections, 15 theorems, 62 equations.

Table of Contents

  1. Introduction
  2. The randomization test
  3. Construction of the randomization test
  4. Framework
  5. One-sample tests on a finite support
  6. One-sample tests on a continuous support
  7. Groups of linear transformations
  8. Arbitrary groups
  9. Conclusion
  10. Examples of randomization tests
  11. Proofs
  12. All proofs excluding the proof of Proposition \ref{['prop:continuous']}
  13. Proof of Proposition \ref{['prop:continuous']}
  14. Definitions
  15. Proof strategy
  16. ...and 2 more sections

Key Result

Theorem 1

Suppose that for any $g \in \mathbb{G}$, $gX \stackrel{d}{=} X$ whenever $X \sim P \in \Omega_0$. Then $\mathop{\mathrm{\mathbb{E}}}\nolimits_P[\phi(X;\mathbb{G})] = \alpha$$\forall \; P \in \Omega_0$. $\blacktriangleleft$$\blacktriangleleft$

Theorems & Definitions (43)

  • Theorem 1
  • proof : Proof of Theorem \ref{['thm:test']}
  • Remark 1
  • Definition 1: Randomization hypothesis
  • Remark 2
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Proposition 4
  • Remark 3
  • ...and 33 more