Cheap Permutation Testing
Carles Domingo-Enrich, Raaz Dwivedi, Lester Mackey
TL;DR
Cheap Permutation Testing introduces a binning strategy to accelerate permutation tests while preserving exact finite-sample false positive control and minimax optimality. By partitioning data into $s$ bins and permuting bin labels, the method computes binwise sufficient statistics that enable test statistics to be updated with cost near that of a single evaluation, yielding runtime $\Theta(\mathcal{B} s^2)$ and memory $\Theta(s^2)$ after an initial $\Theta(c_\phi n^2)$ precomputation. The framework supports homogeneity and independence testing, including MMD/HSIC-type statistics, and provides finite-sample exactness, power guarantees, and minimax optimality results, showing that cheap tests match standard permutation tests in power while dramatically reducing computation. The paper also develops a quantile-comparison technique that tightens finite-sample bounds and demonstrates the approach’s advantage over several baselines (e.g., MMD/HSIC, random Fourier features, and Wilcoxon-Mann-Whitney) in synthetic experiments. Overall, cheap permutation testing offers scalable, exact hypothesis testing with strong theoretical guarantees and practical speedups for large-scale nonparametric tests.
Abstract
Permutation tests are a popular choice for distinguishing distributions and testing independence, due to their exact, finite-sample control of false positives and their minimax optimality when paired with U-statistics. However, standard permutation tests are also expensive, requiring a test statistic to be computed hundreds or thousands of times to detect a separation between distributions. In this work, we offer a simple approach to accelerate testing: group your datapoints into bins and permute only those bins. For U and V-statistics, we prove that these cheap permutation tests have two remarkable properties. First, by storing appropriate sufficient statistics, a cheap test can be run in time comparable to evaluating a single test statistic. Second, cheap permutation power closely approximates standard permutation power. As a result, cheap tests inherit the exact false positive control and minimax optimality of standard permutation tests while running in a fraction of the time. We complement these findings with improved power guarantees for standard permutation testing and experiments demonstrating the benefits of cheap permutations over standard maximum mean discrepancy (MMD), Hilbert-Schmidt independence criterion (HSIC), random Fourier feature, Wilcoxon-Mann-Whitney, cross-MMD, and cross-HSIC tests.