Training Guarantees of Neural Network Classification Two-Sample Tests by Kernel Analysis

TL;DR

This work analyzes neural-network two-sample tests through the neural tangent kernel (NTK) lens, deriving time-to-detection guarantees by examining how the target function $f^* = \frac{p-q}{p+q}$ projects onto the NTK eigenfunctions. By connecting realistic (finite-sample) and population dynamics to a zero-time NTK regime, it establishes approximation bounds (scaling with $t^{3/2}$ and $t^{5/2}$) that preserve monotonic behavior of the test statistic. The main contributions include explicit conditions under which the null and alternative hypotheses are spectrally separated in training time, a comprehensive power analysis, and empirical demonstrations on hard two-sample problems using a multi-layer network. The results offer practical guidance on choosing training time and network complexity to reliably detect distributional differences while controlling false positives. Overall, the paper provides a rigorous framework linking dynamics, spectral properties, and statistical power for neural network-based two-sample testing.

Abstract

We construct and analyze a neural network two-sample test to determine whether two datasets came from the same distribution (null hypothesis) or not (alternative hypothesis). We perform time-analysis on a neural tangent kernel (NTK) two-sample test. In particular, we derive the theoretical minimum training time needed to ensure the NTK two-sample test detects a deviation-level between the datasets. Similarly, we derive the theoretical maximum training time before the NTK two-sample test detects a deviation-level. By approximating the neural network dynamics with the NTK dynamics, we extend this time-analysis to the realistic neural network two-sample test generated from time-varying training dynamics and finite training samples. A similar extension is done for the neural network two-sample test generated from time-varying training dynamics but trained on the population. To give statistical guarantees, we show that the statistical power associated with the neural network two-sample test goes to 1 as the neural network training samples and test evaluation samples go to infinity. Additionally, we prove that the training times needed to detect the same deviation-level in the null and alternative hypothesis scenarios are well-separated. Finally, we run some experiments showcasing a two-layer neural network two-sample test on a hard two-sample test problem and plot a heatmap of the statistical power of the two-sample test in relation to training time and network complexity.

Figures (4)

• Figure 1: Visual for detection levels $t^+(\epsilon)$ and $t^-(\epsilon)$ being well-separated.
• Figure 2: Roadmap of Theory and Results. Blue denotes zero-time NTK auxiliary dynamics, gray denotes population dynamics, and red denotes realistic dynamics.
• Figure 3: Plots statistical power for each epoch and ratio of parameters-to-samples when we are in the alternative hypothesis case with $P$ and $Q$ as above.
• Figure 4: Plots statistical power for each epoch and ratio of parameters-to-samples when we are in the null hypothesis case with samples from $P$.

Theorems & Definitions (63)

• Theorem 1.1: Informal Test Power Analysis
• Theorem 1.2: Informal Test Time Analysis
• Definition 4.2
• Proposition 4.3: Zero-Time Auxiliary Dynamics Solution
• Lemma 4.4: Zero-Time Auxiliary Statistic
• Definition 4.5
• Proposition 4.6: Minimum Detection Time -- Zero-Time Auxiliary
• Remark 4.7
• Proposition 4.8: Maximum Undetectable Time -- Zero-Time Auxiliary
• Corollary 4.9: Detection Times -- Zero-Time Auxiliary