Mathematical Foundations of Neural Tangents and Infinite-Width Networks

TL;DR

The paper develops a rigorous framework for studying neural networks in the finite-width regime via the Neural Tangent Kernel (NTK). It introduces the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), which combines Fourier feature embeddings, layerwise residual scaling, and stochastic depth to manipulate the NTK eigenvalue spectrum and stabilize training. Theoretical contributions include bounds on NTK evolution, connections between eigenvalue spectra and generalization, and analysis of how optimization landscapes evolve with width. Empirical validation on synthetic data, UCI benchmarks, and CIFAR-10 subsets demonstrates improved training stability and generalization, bridging infinite-width theory with practical architectures.

Abstract

We investigate the mathematical foundations of neural networks in the infinite-width regime through the Neural Tangent Kernel (NTK). We propose the NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN), an architecture integrating Fourier feature embeddings, residual connections with layerwise scaling, and stochastic depth to enable rigorous analysis of kernel evolution during training. Our theoretical contributions include deriving bounds on NTK dynamics, characterizing eigenvalue evolution, and linking spectral properties to generalization and optimization stability. Empirical results on synthetic and benchmark datasets validate the predicted kernel behavior and demonstrate improved training stability and generalization. This work provides a comprehensive framework bridging infinite-width theory and practical deep-learning architectures.

Figures (2)

• Figure 1: System architecture of the proposed NTK-Eigenvalue-Controlled Residual Network (NTK-ECRN).
• Figure 2: Evolution of the Neural Tangent Kernel (NTK) over training epochs, showing controlled growth of eigenvalues and stable kernel dynamics.