Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rerandomization for covariate balance mitigates p-hacking in regression adjustment

Xin Lu, Peng Ding

TL;DR

This paper addresses inflation of false discoveries caused by p-hacking in regression-adjusted treatment effect inference by introducing a design-based framework for hacked p-values and proposing two rerandomization schemes, ReM and ReP, to enforce covariate balance in the design stage. It proves that ReM and ReP bound or resolve p-hacking under both sharp and weak null hypotheses, and provides explicit threshold formulas—$\bar{a}_{\text{rem}}(\alpha,R^2_{\boldsymbol{x}},\underline{\Delta})$ for ReM and $\underline{\alpha}_{t}(R^2_{\boldsymbol{x}},\alpha)$ for ReP—that guide practical threshold selection. The results are supported by simulation evidence showing reduced type I error inflation under these schemes, with performance depending on the covariates' predictive power and correlation structure. Collectively, the work offers principled, design-based tools to safeguard regression-adjusted causal inferences against $p$-hacking in randomized experiments and provides actionable guidance for practitioners to choose rerandomization thresholds in practice.

Abstract

Rerandomization enforces covariate balance across treatment groups in the design stage of experiments. Despite its intuitive appeal, its theoretical justification remains unsatisfying because its benefits of improving efficiency for estimating the average treatment effect diminish if we use regression adjustment in the analysis stage. To strengthen the theory of rerandomization, we show that it mitigates false discoveries resulting from $p$-hacking, the practice of strategically selecting covariates to get more significant $p$-values. Moreover, we show that rerandomization with a sufficiently stringent threshold can resolve $p$-hacking. As a byproduct, our theory offers guidance for choosing the threshold in rerandomization in practice.

Rerandomization for covariate balance mitigates p-hacking in regression adjustment

TL;DR

This paper addresses inflation of false discoveries caused by p-hacking in regression-adjusted treatment effect inference by introducing a design-based framework for hacked p-values and proposing two rerandomization schemes, ReM and ReP, to enforce covariate balance in the design stage. It proves that ReM and ReP bound or resolve p-hacking under both sharp and weak null hypotheses, and provides explicit threshold formulas— for ReM and for ReP—that guide practical threshold selection. The results are supported by simulation evidence showing reduced type I error inflation under these schemes, with performance depending on the covariates' predictive power and correlation structure. Collectively, the work offers principled, design-based tools to safeguard regression-adjusted causal inferences against -hacking in randomized experiments and provides actionable guidance for practitioners to choose rerandomization thresholds in practice.

Abstract

Rerandomization enforces covariate balance across treatment groups in the design stage of experiments. Despite its intuitive appeal, its theoretical justification remains unsatisfying because its benefits of improving efficiency for estimating the average treatment effect diminish if we use regression adjustment in the analysis stage. To strengthen the theory of rerandomization, we show that it mitigates false discoveries resulting from -hacking, the practice of strategically selecting covariates to get more significant -values. Moreover, we show that rerandomization with a sufficiently stringent threshold can resolve -hacking. As a byproduct, our theory offers guidance for choosing the threshold in rerandomization in practice.
Paper Structure (30 sections, 30 theorems, 227 equations, 2 figures, 1 table)

This paper contains 30 sections, 30 theorems, 227 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Design-based framework for hacked p-values
  3. Rerandomization mitigates $p$-hacking
  4. ReM mitigates $p$-hacking
  5. Guidance on selecting rerandomization threshold
  6. ReP mitigates $p$-hacking
  7. Simulation study
  8. Discussion
  9. Extension to SRE
  10. Hacked $p$-values under SRE
  11. Rerandomization under SRE
  12. Hacked $p$-values under rerandomization in SRE
  13. Additional notation and lemmas
  14. Asymptotic limits of hacked $p$-values
  15. Joint asymptotic normality for the ATE estimators and difference in means of covariates
  16. ...and 15 more sections

Key Result

Proposition 1

Under $H_{0\textsc{f}}$, we have $\mathbb{P}_{\infty}(p_\textnormal{L}^{\textnormal{h}} \leq \alpha) \geq \alpha$, the equality holds if and only if $R^2_{\boldsymbol{x}}=0$.

Figures (2)

  • Figure 1: Required acceptance probability of ReM to resolve $p$-hacking under different $\alpha$, $R_{\boldsymbol{x}}^2$ and $K$ when \ref{['a:orthogonal-covariates-equally-importance']} holds. The acceptance probability is plotted under the $\log_{10}$ transformation.
  • Figure 2: The plot of $R^2_{\boldsymbol{x}}$ versus empirical type I error rate computed by $10^5$ treatment assignments for different $\rho$ and $\boldsymbol{\beta}$ under different designs. $\boldsymbol{\beta} = \boldsymbol{1}_5$ is labelled as "Equal" and $\boldsymbol{\beta} = (1,1,0.3,0.3,0.3)$ is labelled as "Unequal", respectively. Black dashed lines signify the significance level $\alpha=0.05$. The type I error bounds under different rerandomization schemes, obtained through \ref{['thm:ReM-bound-of-I-error-rate-inflation']} and \ref{['thm:ReP-bound-of-I-error-rate-inflation']}, are shown as dashed lines in colors of the corresponding designs.

Theorems & Definitions (52)

  • Proposition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Corollary 1
  • Theorem 5
  • Theorem 6
  • Theorem S1
  • Theorem S2
  • ...and 42 more