Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A robust and p-hacking-proof significance test under variance uncertainty

Xifeng Li, Shuzhen Yang, Jianfeng Yao

TL;DR

A robust method is proposed for the one-sample significance test that can protect against p-hacking from sample manipulation, and can protect the significance level against potential data manipulation by an experimenter.

Abstract

P-hacking poses challenges to traditional hypothesis testing. In this paper, we propose a robust method for the one-sample significance test that can protect against p-hacking from sample manipulation. Precisely, assuming a sequential arrival of the data whose variance can be time-varying and for which only lower and upper bounds are assumed to exist with possibly unknown values, we use the modern theory of sublinear expectation to build a testing procedure which is robust under such variance uncertainty, and can protect the significance level against potential data manipulation by an experimenter. It is shown that our new method can effectively control the type I error while preserving a satisfactory power, yet a traditional rejection criterion performs poorly under such variance uncertainty. Our theoretical results are well confirmed by a detailed simulation study.

A robust and p-hacking-proof significance test under variance uncertainty

TL;DR

A robust method is proposed for the one-sample significance test that can protect against p-hacking from sample manipulation, and can protect the significance level against potential data manipulation by an experimenter.

Abstract

P-hacking poses challenges to traditional hypothesis testing. In this paper, we propose a robust method for the one-sample significance test that can protect against p-hacking from sample manipulation. Precisely, assuming a sequential arrival of the data whose variance can be time-varying and for which only lower and upper bounds are assumed to exist with possibly unknown values, we use the modern theory of sublinear expectation to build a testing procedure which is robust under such variance uncertainty, and can protect the significance level against potential data manipulation by an experimenter. It is shown that our new method can effectively control the type I error while preserving a satisfactory power, yet a traditional rejection criterion performs poorly under such variance uncertainty. Our theoretical results are well confirmed by a detailed simulation study.

Paper Structure

This paper contains 14 sections, 17 theorems, 65 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The theory of sublinear expectation
  3. The significance test under variance uncertainty
  4. The significance test under variance uncertainty with known variance bounds $\{\underline{\sigma}^{2},\overline{\sigma}^{2}\}$
  5. The significance test under variance uncertainty with unknown variance bounds $\{\underline{\sigma}^{2},\overline{\sigma}^{2}\}$
  6. Performance against p-hacking
  7. An optimal data manipulation strategy for p-hacking
  8. Comparison with the traditional test procedure
  9. Monte-Carlo study
  10. Discussions
  11. Basic concepts of sublinear expectation
  12. Two variants of the significance test
  13. First variant of the significance test
  14. Second variant of the significance test

Key Result

Theorem 3.1

Let $\{X_{i}\}_{i=1}^{n}$ be given by the data generating process [*] above. Then for any Lipschitz function $\varphi$, where $\{u(t,x; \varphi):(t,x)\in [0,\infty)\times\mathbb{R}\}$ is the unique viscosity solution to the Cauchy problem, In the above expression, $u_t=\partial u/\partial t$, $u_{xx}=\partial^{2} u/\partial x^{2}$, and $a^+$ and $a^-$ denote the positive and negative parts of $a

Figures (4)

  • Figure 1: Empirical type I error rate ($\mu=\mu_{0}$) plot over 5,000 repetitions for Simulation \ref{['sim:1']}
  • Figure 2: Empirical power plots over 5,000 repetitions for Simulation \ref{['sim:1']} with n=100 and varying $\mu$ (left panel), and with $\mu$=0.1 and varying $n$ (right panel).
  • Figure 3: Empirical type I error rate ($\mu=\mu_{0}$) plot over 5,000 repetitions for Simulation \ref{['sim:2']}
  • Figure 4: Empirical power plots over 5,000 repetitions for Simulation \ref{['sim:2']} with n=100 and varying $\mu$ (left panel), and with $\mu$=0.1 and varying $n$ (right panel).

Theorems & Definitions (42)

  • Remark 3.1
  • Theorem 3.1
  • Remark 3.2
  • Lemma 3.1
  • proof
  • Remark 3.3
  • Theorem 3.2
  • proof
  • Corollary 3.1
  • proof
  • ...and 32 more