Approximation Error and Complexity Bounds for ReLU Networks on Low-Regular Function Spaces

TL;DR

This work studies the approximation of a broad class of bounded, low-regularity functions $f$ with integrable Fourier transform by ReLU networks. The authors develop a constructive approach that leverages Fourier features residual networks (FFNets) and a broad toolkit of special ReLU networks to obtain explicit error and complexity bounds, showing that the $L^2$ approximation error can be controlled by a quantity proportional to $||f||_{\infty}$ and inversely proportional to the product of width and depth, $WL$. The main contributions include a detailed bound $WL \le C d \frac{||f||_{L^{\infty}(\Theta)}^2}{\varepsilon^2}\left(1+\ln\left(\frac{||\hat f||_{L^{1}}}{||f||_{\infty}}\right)\right)^2 \log_2^2(\varepsilon^{-1})$ for univariate inputs (extendable to multi-dimensions) and a corresponding approximation-error rate with a tunable exponent $\alpha(\varepsilon)$, derived via a constructive approximation of FFNets by ReLU nets. The results provide concrete, implementable guidelines for building ReLU networks that provably approximate low-regularity targets and quantify the trade-off between network complexity and approximation quality.

Abstract

In this work, we consider the approximation of a large class of bounded functions, with minimal regularity assumptions, by ReLU neural networks. We show that the approximation error can be bounded from above by a quantity proportional to the uniform norm of the target function and inversely proportional to the product of network width and depth. We inherit this approximation error bound from Fourier features residual networks, a type of neural network that uses complex exponential activation functions. Our proof is constructive and proceeds by conducting a careful complexity analysis associated with the approximation of a Fourier features residual network by a ReLU network.

Figures (12)

• Figure 1: An example of a target function in $S$
• Figure 2: ReLU network $\Psi=\{(M^{(\ell)},\bm{b}^{(\ell))}\}_{\ell=0}^{3}$ realizing a function $f_{\Psi}\in \mathcal{N}^{D_{1},D_{2},1}_{3,3}$
• Figure 3: Graphical representation of Fourier features residual network with $W_{FF}=1$ and $L_{FF}=3$ approximating a uni-variate target function $f$
• Figure 4: Graph representations of type 1 (left) and type 2 (right) special ReLU networks in $\tilde{\mathcal{N}}^{D_{1},D_{2},1}_{4,3}$. The source channels are highlighted in blue while the collation channels are highlighted in red.
• Figure 5: Extending network depth using identity mapping hidden layers

Theorems & Definitions (33)

• Remark 2.1
• theorem 1: Main Complexity Result
• Remark 3.1
• theorem 2: Main approximation error result
• proof
• theorem 3: Approximation error in Fourier features residual network kammonen2023smaller
• Lemma 1
• proof
• Remark 4.1
• Lemma 2