Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces

TL;DR

This work analyzes how efficiently deep ReLU networks can approximate Sobolev spaces $W^s(L_q)$ and Besov spaces $B^s_r(L_q)$ on the unit cube, with error measured in $L_p$, for all $1≤p,q≤∞$ under the compact embedding condition $\frac{1}{q}-\frac{1}{p}<\frac{s}{d}$. The authors introduce a novel bit-extraction based sparse encoding to achieve sharp (super-convergent) rates in the nonlinear regime $p>q$ and develop VC-dimension based lower bounds to prove rate optimality; they provide explicit upper bounds with fixed width and growing depth, and show the Besov case mirrors the Sobolev results. The key results include $\inf_{f_L\in\Upsilon^{25d+31,L}(\mathbb{R}^d)} \|f-f_L\|_{L_p(Ω)} \le C\|f\|_{W^s(L_q(Ω))} L^{-2s/d}$ for Sobolev functions and an analogous bound for Besov functions, highlighting the potential of very deep networks to outperform classical approximation methods in parameter efficiency, albeit with non-encodable parameter representations. Overall, the paper advances the theoretical understanding of nonlinear approximation by deep ReLU networks in clinically relevant function spaces, with implications for scientific computing and high-dimensional approximation.

Abstract

Let $Ω= [0,1]^d$ be the unit cube in $\mathbb{R}^d$. We study the problem of how efficiently, in terms of the number of parameters, deep neural networks with the ReLU activation function can approximate functions in the Sobolev spaces $W^s(L_q(Ω))$ and Besov spaces $B^s_r(L_q(Ω))$, with error measured in the $L_p(Ω)$ norm. This problem is important when studying the application of neural networks in a variety of fields, including scientific computing and signal processing, and has previously been solved only when $p=q=\infty$. Our contribution is to provide a complete solution for all $1\leq p,q\leq \infty$ and $s > 0$ for which the corresponding Sobolev or Besov space compactly embeds into $L_p$. The key technical tool is a novel bit-extraction technique which gives an optimal encoding of sparse vectors. This enables us to obtain sharp upper bounds in the non-linear regime where $p > q$. We also provide a novel method for deriving $L_p$-approximation lower bounds based upon VC-dimension when $p < \infty$. Our results show that very deep ReLU networks significantly outperform classical methods of approximation in terms of the number of parameters, but that this comes at the cost of parameters which are not encodable.

Theorems & Definitions (45)

• Theorem 1
• Theorem 2
• Theorem 3
• Lemma 1: Composing Networks
• Lemma 2: Concatenating Networks
• proof
• Lemma 3
• Proposition 1: Summing Networks
• Proposition 2
• Proposition 3: Product Network, Proposition 3 in yarotsky2017error