Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonlocal techniques for the analysis of deep ReLU neural network approximations

Cornelia Schneider, Mario Ullrich, Jan Vybiral

TL;DR

This work extends a nonlocal Riesz-basis framework built from the piecewise-linear systems ${\mathcal C}_k$ and ${\mathcal S}_k$ to Sobolev spaces $W^s([0,1]^d)$ and Barron-type spaces ${\mathbb B}^s([0,1]^d)$ for $0< s<1$, with constants that are independent of dimension. It proves that properly scaled basis elements yield Riesz bases in these spaces and provides explicit neural-network constructions that reproduce finite basis sums exactly, achieving $L_2$-approximation error $\varepsilon$ without suffering from the curse of dimensionality in several regimes. The paper then develops approximation results for deep ReLU networks in both Sobolev and Barron spaces, including fixed-architecture guarantees and best $n$-term strategies, and addresses function-value sampling via least-squares and basis-pursuit-denoising, establishing NN-sampling bounds that link nonlocal representations to data-driven recovery. Collectively, the results offer dimension-aware, nonlocal methods for neural-network approximation and recovery in high dimensions, with explicit network architectures and quantitative error/divergence bounds.

Abstract

Recently, Daubechies, DeVore, Foucart, Hanin, and Petrova introduced a system of piece-wise linear functions, which can be easily reproduced by artificial neural networks with the ReLU activation function and which form a Riesz basis of $L_2([0,1])$. This work was generalized by two of the authors to the multivariate setting. We show that this system serves as a Riesz basis also for Sobolev spaces $W^s([0,1]^d)$ and Barron classes ${\mathbb B}^s([0,1]^d)$ with smoothness $0<s<1$. We apply this fact to re-prove some recent results on the approximation of functions from these classes by deep neural networks. Our proof method avoids using local approximations and allows us to track also the implicit constants as well as to show that we can avoid the curse of dimension. Moreover, we also study how well one can approximate Sobolev and Barron functions by ANNs if only function values are known.

Nonlocal techniques for the analysis of deep ReLU neural network approximations

TL;DR

This work extends a nonlocal Riesz-basis framework built from the piecewise-linear systems and to Sobolev spaces and Barron-type spaces for , with constants that are independent of dimension. It proves that properly scaled basis elements yield Riesz bases in these spaces and provides explicit neural-network constructions that reproduce finite basis sums exactly, achieving -approximation error without suffering from the curse of dimensionality in several regimes. The paper then develops approximation results for deep ReLU networks in both Sobolev and Barron spaces, including fixed-architecture guarantees and best -term strategies, and addresses function-value sampling via least-squares and basis-pursuit-denoising, establishing NN-sampling bounds that link nonlocal representations to data-driven recovery. Collectively, the results offer dimension-aware, nonlocal methods for neural-network approximation and recovery in high dimensions, with explicit network architectures and quantitative error/divergence bounds.

Abstract

Recently, Daubechies, DeVore, Foucart, Hanin, and Petrova introduced a system of piece-wise linear functions, which can be easily reproduced by artificial neural networks with the ReLU activation function and which form a Riesz basis of . This work was generalized by two of the authors to the multivariate setting. We show that this system serves as a Riesz basis also for Sobolev spaces and Barron classes with smoothness . We apply this fact to re-prove some recent results on the approximation of functions from these classes by deep neural networks. Our proof method avoids using local approximations and allows us to track also the implicit constants as well as to show that we can avoid the curse of dimension. Moreover, we also study how well one can approximate Sobolev and Barron functions by ANNs if only function values are known.

Paper Structure

This paper contains 12 sections, 14 theorems, 130 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Riesz bases of Sobolev and Barron classes
  3. Univariate Sobolev classes
  4. Multivariate Sobolev classes
  5. Barron classes
  6. Approximation of functions by ANNs
  7. Sobolev classes
  8. Barron classes
  9. Discussion
  10. Sobolev spaces
  11. Barron spaces
  12. Approximation with ANNs using function values

Key Result

Theorem 1

Let $0\le s<1$. Then, $W^s([0,1])=\mathcal{F}^s([0,1])$ in the sense of equivalent norms. Moreover, the system is a Riesz basis of $W^s([0,1])$ and for constants $c,C>0$ it holds Modifications of ineq-Riesz-1d for functions with constant term are obvious (by incorporating $\alpha_0$).

Figures (5)

  • Figure 1: Feed-forward $\mathrm{ReLU}$ network with length $L$ and width $W$
  • Figure 2: ANN representing $H$ and usual graph associated with $H$
  • Figure 3: ANN representing the sum of functions $\mathcal{N}_1, \ldots, \mathcal{N}_N$
  • Figure 4: Illustration of $N(t,2)$
  • Figure 5: Visualization of \ref{['help']}, and splitting of $Z(t,2)$ into disks with $|k_1|\le\lfloor t\rfloor$

Theorems & Definitions (33)

  • Theorem 1
  • proof
  • Remark 1
  • Remark 2
  • Theorem 2
  • proof : Proof of Theorem \ref{['thm:FsWs_d']}.
  • Theorem 3
  • proof
  • Remark 3
  • Remark 4
  • ...and 23 more