Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

Jonathan W. Siegel, Jinchao Xu

TL;DR

The paper addresses the fundamental limits of non-linear approximation by shallow neural networks through the lens of metric entropy and $n$-widths of associated variation spaces. It introduces smoothly parameterized dictionaries and leverages Maurey’s sampling to obtain sharp upper bounds on approximation rates $\|f-f_n\|_X \lesssim n^{-1/2-s/d}$, with $s$ determined by the dictionary’s smooth parameterization; applied to ReLU$^k$ dictionaries, this yields $L^p$-rates $\lesssim n^{-1/2-(pk+1)/(pd)}$. Complementarily, it develops lower-bound techniques for ridge-function dictionaries, establishing sharp decay rates for entropy, Kolmogorov, and Bernstein widths for $B_1(\mathbb{D})$, and proving that the shallow ReLU$^k$ approximation rate exponents $\frac{1}{2d}$ and $\frac{2k+1}{2d}$ cannot be improved in general. By matching upper and lower bounds (notably in the $p=2$ setting) the work clarifies the exact scaling of approximation capacity with dimension and network width, and it characterizes the gap between shallow nets and Barron/Radon BV frameworks. These results illuminate the intrinsic limits of non-linear, dictionary-based approximation for neural networks and provide a rigorous basis for understanding when shallow architectures achieve optimal rates. $\,$

Abstract

In this article, we study approximation properties of the variation spaces corresponding to shallow neural networks with a variety of activation functions. We introduce two main tools for estimating the metric entropy, approximation rates, and $n$-widths of these spaces. First, we introduce the notion of a smoothly parameterized dictionary and give upper bounds on the non-linear approximation rates, metric entropy and $n$-widths of their absolute convex hull. The upper bounds depend upon the order of smoothness of the parameterization. This result is applied to dictionaries of ridge functions corresponding to shallow neural networks, and they improve upon existing results in many cases. Next, we provide a method for lower bounding the metric entropy and $n$-widths of variation spaces which contain certain classes of ridge functions. This result gives sharp lower bounds on the $L^2$-approximation rates, metric entropy, and $n$-widths for variation spaces corresponding to neural networks with a range of important activation functions, including ReLU$^k$ activation functions and sigmoidal activation functions with bounded variation.

Sharp Bounds on the Approximation Rates, Metric Entropy, and $n$-widths of Shallow Neural Networks

TL;DR

The paper addresses the fundamental limits of non-linear approximation by shallow neural networks through the lens of metric entropy and -widths of associated variation spaces. It introduces smoothly parameterized dictionaries and leverages Maurey’s sampling to obtain sharp upper bounds on approximation rates , with determined by the dictionary’s smooth parameterization; applied to ReLU dictionaries, this yields -rates . Complementarily, it develops lower-bound techniques for ridge-function dictionaries, establishing sharp decay rates for entropy, Kolmogorov, and Bernstein widths for , and proving that the shallow ReLU approximation rate exponents and cannot be improved in general. By matching upper and lower bounds (notably in the setting) the work clarifies the exact scaling of approximation capacity with dimension and network width, and it characterizes the gap between shallow nets and Barron/Radon BV frameworks. These results illuminate the intrinsic limits of non-linear, dictionary-based approximation for neural networks and provide a rigorous basis for understanding when shallow architectures achieve optimal rates.

Abstract

In this article, we study approximation properties of the variation spaces corresponding to shallow neural networks with a variety of activation functions. We introduce two main tools for estimating the metric entropy, approximation rates, and -widths of these spaces. First, we introduce the notion of a smoothly parameterized dictionary and give upper bounds on the non-linear approximation rates, metric entropy and -widths of their absolute convex hull. The upper bounds depend upon the order of smoothness of the parameterization. This result is applied to dictionaries of ridge functions corresponding to shallow neural networks, and they improve upon existing results in many cases. Next, we provide a method for lower bounding the metric entropy and -widths of variation spaces which contain certain classes of ridge functions. This result gives sharp lower bounds on the -approximation rates, metric entropy, and -widths for variation spaces corresponding to neural networks with a range of important activation functions, including ReLU activation functions and sigmoidal activation functions with bounded variation.

Paper Structure

This paper contains 15 sections, 27 theorems, 271 equations.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Prior Results
  4. Main Results
  5. Type-2 Spaces and Maurey's Sampling Argument
  6. Smoothly parameterized dictionaries
  7. Polynomial Interpolation Error Bounds
  8. Approximation rates for smoothly parameterized dictionaries
  9. Metric entropy bounds for smoothly parameterized dictionaries
  10. Kolmogorov $n$-width bounds for smoothly parameterized dictionaries
  11. Gelfand numbers of smoothly parameterized dictionaries
  12. Lower Bounds for Dictionaries of Ridge Functions
  13. Lower bounds on approximation rates for shallow neural networks
  14. Conclusion
  15. Acknowledgements

Key Result

Theorem 1

Suppose that $X$ is a type-2 Banach space and $\mathbb{D}\subset X$ is a dictionary with $K_\mathbb{D}:=\sup_{d\in \mathbb{D}}\|d\|_X < \infty$. Then for $f\in B_1(\mathbb{D})$, we have

Theorems & Definitions (52)

  • Definition 1
  • Theorem 1
  • proof
  • Definition 2
  • Definition 3
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • ...and 42 more