Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

ZeYu Li, FengLei Fan, TieYong Zeng

Abstract

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and $L^p$ functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general $L^p$ functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.

Efficient Approximation to Analytic and $L^p$ functions by Height-Augmented ReLU Networks

Abstract

This work addresses two fundamental limitations in neural network approximation theory. We demonstrate that a three-dimensional network architecture enables a significantly more efficient representation of sawtooth functions, which serves as the cornerstone in the approximation of analytic and functions. First, we establish substantially improved exponential approximation rates for several important classes of analytic functions and offer a parameter-efficient network design. Second, for the first time, we derive a quantitative and non-asymptotic approximation of high orders for general functions. Our techniques advance the theoretical understanding of the neural network approximation in fundamental function spaces and offer a theoretically grounded pathway for designing more parameter-efficient networks.
Paper Structure (8 sections, 20 theorems, 139 equations, 5 figures, 1 table)

This paper contains 8 sections, 20 theorems, 139 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Notation and Definition
  3. Approximation of Analytic Functions
  4. Approximation to Polynomials and $C^{\infty}$ Functions
  5. Approximation of Analytic Functions on the Cube
  6. Approximation of Analytic Functions on $L^2(\mathbb{R}^d,\gamma_d)$
  7. Approximation of $L^p$ Functions
  8. Conclusion

Key Result

Lemma 3.1

For any $H>0$, there is a 3D ReLU network $\mathcal{N}_{2,1,H}$, whose function $f_H: [0,1]\to [0,1]$ satisfies

Figures (5)

  • Figure 1: Adding intra-layer links creates a new hierarchy among neurons in the same layer, which induces a new dimension referred to as height. Note that a 2D network can be regarded as a 3D network with height=1.
  • Figure 2: Implementing $f_H$ by 3D networks in \ref{['prop_square']}.
  • Figure 3: Illustration of the construction in \ref{['approx_poly']}
  • Figure 4: Illustration of the network structure in \ref{['multi_prod']}
  • Figure 5: Implementation of the network approximation to analytic functions in \ref{['holomorphic_R^d']}.

Theorems & Definitions (44)

  • Definition 2.1: Real analytic functions
  • Definition 2.2: Holomorphic functions
  • Definition 2.3
  • Definition 2.4: $L^p$ modulus of smoothness
  • Definition 2.5: 3D networks Fan2025Expressivity
  • Definition 2.6
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • ...and 34 more