Sobolev Approximation of Deep ReLU Network in Log-weighted Barron Space

TL;DR

This paper addresses the difficulty of explaining why deep ReLU networks approximate high-dimensional functions efficiently by introducing the log-Barron space $\mathscr{B}^{\log}$, a Banach space with weaker spectral decay than classical Barron spaces. It establishes fundamental embedding relations with Sobolev spaces and a bound on the Rademacher complexity, showing that deep, width-bounded ReLU networks can achieve dimension-independent approximation in $L^2$ and $H^1$ for functions in $\mathscr{B}^{\log}$; depth, rather than width, governs the regularity requirements via the norms $\|f\|_{\mathscr{B}^{\log}}$ and $\|f\|_{\mathscr{B}^0}$. The results are extended to a Sobolev setting with the enriched space $\mathscr{B}^{s,\log}$, providing explicit depth bounds and rates for $L^2$ and $H^1$ convergence, and illustrating how depth mitigates smoothness assumptions. Overall, the work offers a rigorous mechanism by which depth contributes to the expressive power of neural nets in high dimensions, clarifying their robust performance on complex, high-frequency signals and informing future function-space analyses of deep learning models.

Abstract

Universal approximation theorems show that neural networks can approximate any continuous function; however, the number of parameters may grow exponentially with the ambient dimension, so these results do not fully explain the practical success of deep models on high-dimensional data. Barron space theory addresses this: if a target function belongs to a Barron space, a two-layer network with $n$ parameters achieves an $O(n^{-1/2})$ approximation error in $L^2$. Yet classical Barron spaces $\mathscr{B}^{s+1}$ still require stronger regularity than Sobolev spaces $H^s$, and existing depth-sensitive results often assume constraints such as $sL \le 1/2$. In this paper, we introduce a log-weighted Barron space $\mathscr{B}^{\log}$, which requires a strictly weaker assumption than $\mathscr{B}^s$ for any $s>0$. For this new function space, we first study embedding properties and carry out a statistical analysis via the Rademacher complexity. Then we prove that functions in $\mathscr{B}^{\log}$ can be approximated by deep ReLU networks with explicit depth dependence. We then define a family $\mathscr{B}^{s,\log}$, establish approximation bounds in the $H^1$ norm, and identify maximal depth scales under which these rates are preserved. Our results clarify how depth reduces regularity requirements for efficient representation, offering a more precise explanation for the performance of deep architectures beyond the classical Barron setting, and for their stable use in high-dimensional problems used today.

Figures (4)

• Figure 1: (Left) Distribution of Fourier spectra of the target function $f$. (Right) Approximation error of a deep neural network for various input dimensions $d$ and depth. The target function is defined by $f\left(x\right)=\frac{1}{N}\sum_{i=1}^n \left\vert\hat{f}\left(\xi_i\right)\right\vert e^{2\pi i \xi\cdot x}$. This illustrates the necessity of establishing a dimension-independent approximation theory with respect to depth.
• Figure 2: Graph of $\gamma\left(\cdot,r\right)$ and $\gamma\left(t,\cdot\right)$.
• Figure 3: Construction of $\beta$, $\gamma\left(\cdot,r\right)$ and $F\left(x;\xi_i,r_i\right)$. $\sigma$ denotes $\operatorname{ReLU}$ activation function.
• Figure 4: Network construction for Theorem \ref{['thm:L2_convergence_unit']}(left) and Corollary \ref{['cor:L2_convergence_unit']}(right).

Theorems & Definitions (22)

• Lemma 3.1
• Lemma 3.2: telgarsky2016benefitsliao2025spectral
• Proposition 4.1
• Lemma 4.2: meng2022newliao2025spectral
• Corollary 4.3
• Proposition 4.4
• Proposition 4.5
• Theorem 4.6
• Proposition 4.7: Dudley's entropy integral
• Theorem 4.8: Bound on the Rademacher complexity