Integral Representations of Sobolev Spaces via ReLU$^k$ Activation Function and Optimal Error Estimates for Linearized Networks

TL;DR

This work develops a foundational link between Sobolev regularity and neural network representations by proving that Sobolev spaces $H^{\frac{d+2k+1}{2}}(\Omega)$ admit an integral representation via ReLU$^k$ ridge functions on the sphere, and that linearized shallow networks achieve the optimal rate $\mathcal{O}\big(n^{- frac{1}{2}-\tfrac{2k+1}{2d}}\big)$ in these spaces. It further shows that $\mathcal{H}^{\frac{d+2k+1}{2}}(\Omega)$ coincides with the RKHS induced by the kernel $K_{\text{ReLU}^k}$, and provides a tight comparison with Barron spaces $\mathcal{B}^k(\Omega)$. The analysis spans sphere and ball geometries, using spherical harmonics, Legendre expansions, and well-distributed point sets to obtain sharp approximation rates for finite-neuron spaces $L_{n,M}^k$, with coefficient bounds crucial for generalization. The work also connects deterministic and randomized approaches, showing that random features achieve near-optimal rates up to logarithmic factors, while deterministic well-distributed weights attain similar objectives without probabilistic failure. Together, these results offer a rigorous foundation for the expressive power and generalization of linearized ReLU$^k$ networks in Sobolev settings and provide a framework for kernel- and RKHS-based interpretations of Sobolev function approximation by neural-inspired architectures.

Abstract

This paper presents two main theoretical results concerning shallow neural networks with ReLU$^k$ activation functions. We establish a novel integral representation for Sobolev spaces, showing that every function in $H^{\frac{d+2k+1}{2}}(Ω)$ can be expressed as an $L^2$-weighted integral of ReLU$^k$ ridge functions over the unit sphere. This result mirrors the known representation of Barron spaces and highlights a fundamental connection between Sobolev regularity and neural network representations. Moreover, we prove that linearized shallow networks -- constructed by fixed inner parameters and optimizing only the linear coefficients -- achieve optimal approximation rates $O(n^{-\frac{1}{2}-\frac{2k+1}{2d}})$ in Sobolev spaces.

Theorems & Definitions (35)

• Theorem 2.1: siegel2022sharpsiegel2023characterization
• Definition 2.1: Well-distributed and quasi-uniform
• Theorem 2.2
• Remark 2.1
• Corollary 2.1: Theorem 8.2. petrushev1998approximation
• proof
• Corollary 2.2: Theorem 1. mao2024approximation
• proof
• Theorem 2.3
• Theorem 2.4