Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis

Jian Lu, Xiaohuang Huang

TL;DR

This paper analyzes the approximation capabilities of shallow neural networks under Barron-type and Sobolev regularity, focusing on activation functions that are powers of exponentials and on the ReLU$^{k}$ family. Using Fourier-based representations and compact dictionary constructions, it derives upper bounds for approximation errors in $H^{m}$, $L^{p}$, and Sobolev norms, elucidating how the dimension $d$ and smoothness interact with activation properties. It establishes that exponential-power activations can yield improved rates in Barron spaces (Theorem 1) while proving that, under $\ell^{1}$-coefficient constraints or insufficient smoothness, the optimal rate $O(n^{m-(k+1)})$ for ReLU$^{k}$ networks is unattainable (Theorems 2–3). The work further characterizes the intrinsic complexity of function classes via $n$-widths and metric entropies in $L^{p}$ and Sobolev spaces (Theorems 6–9, 4–5, Propositions 1–2), confirming the curse of dimensionality and delivering rate-optimal bounds that inform activation selection and network design for high-dimensional approximation tasks.

Abstract

This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness of the function being approximated within the Barron function space. We examine the approximation rates of ReLU$^{k}$ activation functions, proving that the optimal rate cannot be achieved under $\ell^{1}$-bounded coefficients or insufficient smoothness conditions. We also establish optimal approximation rates in various norms for functions in Barron spaces and Sobolev spaces, confirming the curse of dimensionality. Our results clarify the limits of shallow neural networks' approximation capabilities and offer insights into the selection of activation functions and network structures.

Approximation Rates of Shallow Neural Networks: Barron Spaces, Activation Functions and Optimality Analysis

TL;DR

This paper analyzes the approximation capabilities of shallow neural networks under Barron-type and Sobolev regularity, focusing on activation functions that are powers of exponentials and on the ReLU family. Using Fourier-based representations and compact dictionary constructions, it derives upper bounds for approximation errors in , , and Sobolev norms, elucidating how the dimension and smoothness interact with activation properties. It establishes that exponential-power activations can yield improved rates in Barron spaces (Theorem 1) while proving that, under -coefficient constraints or insufficient smoothness, the optimal rate for ReLU networks is unattainable (Theorems 2–3). The work further characterizes the intrinsic complexity of function classes via -widths and metric entropies in and Sobolev spaces (Theorems 6–9, 4–5, Propositions 1–2), confirming the curse of dimensionality and delivering rate-optimal bounds that inform activation selection and network design for high-dimensional approximation tasks.

Abstract

This paper investigates the approximation properties of shallow neural networks with activation functions that are powers of exponential functions. It focuses on the dependence of the approximation rate on the dimension and the smoothness of the function being approximated within the Barron function space. We examine the approximation rates of ReLU activation functions, proving that the optimal rate cannot be achieved under -bounded coefficients or insufficient smoothness conditions. We also establish optimal approximation rates in various norms for functions in Barron spaces and Sobolev spaces, confirming the curse of dimensionality. Our results clarify the limits of shallow neural networks' approximation capabilities and offer insights into the selection of activation functions and network structures.
Paper Structure (7 sections, 218 equations)

This paper contains 7 sections, 218 equations.

Table of Contents

  1. Introduction
  2. The Proof of Theorem 1
  3. Counterexamples for Shallow Neural Networks with ReLU$^k$ Activation Functions
  4. The Proof of Theorem 2
  5. The Proof of Theorem 3
  6. $n$-widths of $\mathbb{P}_{k}^{d}$ in $L^{p}$ and Sobolev space
  7. Conclusion

Theorems & Definitions (10)

  • proof
  • proof
  • proof
  • proof
  • proof
  • proof
  • proof
  • proof
  • proof
  • proof