Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spheres

TL;DR

We study the conditioning of the mass matrix $M$ and stiffness matrix $K_s$ arising from shallow ReLU$^k$ networks on the sphere $\mathbb{S}^d$, proving sharp eigenvalue asymptotics under antipodally quasi-uniform designs. The approach blends spherical-harmonic expansions, Legendre coefficient decay, and a needlet-type block decomposition to obtain precise spectral bounds: $\lambda_j(M) \sim n j^{-(d+2k+1)/d}$ and $\lambda_j(K_s) \sim n j^{-(d+2(k-s)+1)/d}$, leading to conditioning scalings $\kappa(M) \sim n^{(d+2k+1)/d}$ and $\kappa(K_s) \sim n^{(d+2(k-s)+1)/d}$. The analysis leverages a quadrature framework on the sphere and a detailed localization of needlet blocks, relating the spectrum of the discrete Gram matrices to the continuous kernel operators and enabling Weyl-type eigensize counts and effective degrees of freedom results. Numerically, quasi-uniform point sets yield stable conditioning with polynomial dependence on $n$, while random feature samplings can degrade stability due to smaller separation distances, highlighting a trade-off between simplicity and numerical robustness. These results ground the understanding of stability-approximation trade-offs in spectral shallow networks and suggest avenues for preconditioning and geometric design in high-dimensional learning on manifolds.

Abstract

We present an estimation of the condition numbers of the \emph{mass} and \emph{stiffness} matrices arising from shallow ReLU$^k$ neural networks defined on the unit sphere~$\mathbb{S}^d$. In particular, when $\{θ_j^*\}_{j=1}^n \subset \mathbb{S}^d$ is \emph{antipodally quasi-uniform}, the condition number is sharp. Indeed, in this case, we obtain sharp asymptotic estimates for the full spectrum of eigenvalues and characterize the structure of the corresponding eigenspaces, showing that the smallest eigenvalues are associated with an eigenbasis of low-degree polynomials while the largest eigenvalues are linked to high-degree polynomials. This spectral analysis establishes a precise correspondence between the approximation power of the network and its numerical stability.

Figures (1)

• Figure 1: Condition number $\kappa(M)$ of the mass matrix versus the number of neurons $n$ for different dimensions $d$ and activation orders $k$. The dashed lines represent the theoretical growth rate $\mathcal{O}(n^{(d+2k+1)/d})$. The empirical results show excellent agreement with the theory.

Theorems & Definitions (22)

• Definition 1: Antipodally quasi-uniform
• Lemma 1: Lemma 3.1, liu2025achieving
• Lemma 2
• proof
• Theorem 1
• Remark 1
• Lemma 3: mhaskar2001spherical, narcowich2006sobolev
• Theorem 2
• Remark 2
• Corollary 1: Effective dimension