Scale-Insensitive Neural Network Significance Tests

TL;DR

This paper develops a scale-insensitive framework for neural network significance testing that generalizes prior work by replacing metric entropy with Rademacher complexity, allowing unbounded weights and general Lipschitz activations. It weakens target regularity to $f_\star\in H^s([-1,1]^d)$ with $s> d/2$, ensuring Sobolev embeddings yield continuity while preserving optimal approximation rates. A moment-based sieve construction replaces weight-constrained sieves, and localization arguments together with a functional delta method yield consistent estimators and an asymptotic distribution for a covariate-significance statistic based on squared partial derivatives. The framework provides a discretization scheme for practical distributional approximation and proposes adaptive, robust implementations suitable for modern deep networks, with potential extensions to broader architectures and data structures. Overall, the work bridges theory and practice by aligning deep learning statistical guarantees with unconstrained optimization and broad activation functions, while maintaining rigorous nonparametric convergence properties and valid inference.

Abstract

This paper develops a scale-insensitive framework for neural network significance testing, substantially generalizing existing approaches through three key innovations. First, we replace metric entropy calculations with Rademacher complexity bounds, enabling the analysis of neural networks without requiring bounded weights or specific architectural constraints. Second, we weaken the regularity conditions on the target function to require only Sobolev space membership $H^s([-1,1]^d)$ with $s > d/2$, significantly relaxing previous smoothness assumptions while maintaining optimal approximation rates. Third, we introduce a modified sieve space construction based on moment bounds rather than weight constraints, providing a more natural theoretical framework for modern deep learning practices. Our approach achieves these generalizations while preserving optimal convergence rates and establishing valid asymptotic distributions for test statistics. The technical foundation combines localization theory, sharp concentration inequalities, and scale-insensitive complexity measures to handle unbounded weights and general Lipschitz activation functions. This framework better aligns theoretical guarantees with contemporary deep learning practice while maintaining mathematical rigor.

Figures (1)

• Figure 1: Architecture of a fully connected feed-forward neural network with two hidden layers (MLP). Each node in a layer is connected to all nodes in the subsequent layer, with weights characterizing the strength of these connections. The hidden nodes apply an activation function to their inputs.

Theorems & Definitions (37)

• Remark 1
• Definition 1: Neural Network Class
• Definition 2: Empirical Rademacher Complexity
• Proposition 1: Sufficient Conditions
• Theorem 1: Scale-Insensitive Consistency
• Proposition 2: Sieve Properties
• Theorem 2: Asymptotic Distribution
• Theorem 3: Distribution of Test Statistic
• Theorem 4: Asymptotic Distribution of $\frac{1}{n}\sum_{i=1}^n \left(\frac{\partial \hat{f}_n}{\partial x_j}(X_i)\right)^2$
• proof