Residual permutation test for regression coefficient testing
Kaiyue Wen, Tengyao Wang, Yuhao Wang
TL;DR
This work introduces the Residual Permutation Test (RPT) for testing a single regression coefficient in a fixed-design linear model $\boldsymbol{Y}=\boldsymbol{X}\beta+b\boldsymbol{Z}+\boldsymbol{\varepsilon}$ with $p$ allowed to grow proportionally with $n$, achieving finite-sample size control under exchangeable noise when $p<n/2$. RPT builds a permutation-based p-value by projecting residuals onto the intersection of subspaces associated with the original and permuted designs, and proves uniform validity of the p-value under the null without requiring Gaussianity. It further analyzes statistical power under heavy-tailed noise with finite $(1+t)$-th moments, showing detectability when $|b| \ge c n^{-t/(1+t)}$ (or $|b|=\omega(n^{-1/2})$ if $t=1$), and proves these rates are essentially minimax-optimal. Numerical experiments corroborate finite-sample validity and demonstrate competitive power across a range of noise tails, with a variant RPT$_{EM}$ offering improved performance in some heavy-tailed scenarios. The paper also discusses extensions to diverging permutation sets, broader alternatives, and nonlinear relations, providing a comprehensive theoretical and empirical framework for robust coefficient testing in high-dimensional linear models.
Abstract
We consider the problem of testing whether a single coefficient is equal to zero in linear models when the dimension of covariates $p$ can be up to a constant fraction of sample size $n$. In this regime, an important topic is to propose tests with finite-sample valid size control without requiring the noise to follow strong distributional assumptions. In this paper, we propose a new method, called residual permutation test (RPT), which is constructed by projecting the regression residuals onto the space orthogonal to the union of the column spaces of the original and permuted design matrices. RPT can be proved to achieve finite-population size validity under fixed design with just exchangeable noises, whenever $p < n / 2$. Moreover, RPT is shown to be asymptotically powerful for heavy tailed noises with bounded $(1+t)$-th order moment when the true coefficient is at least of order $n^{-t/(1+t)}$ for $t \in [0,1]$. We further proved that this signal size requirement is essentially rate-optimal in the minimax sense. Numerical studies confirm that RPT performs well in a wide range of simulation settings with normal and heavy-tailed noise distributions.