Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniform Approximation with Quadratic Neural Networks

Ahmed Abdeljawad

TL;DR

It is constructively proved that deep neural networks with ReQU activation can approximate any function within the \(R\)-ball of \(r\)-Holder-regular functions up to any accuracy up to any accuracy \(\epsilon \) with at most \(\mathcal{O}\left(\epsilon^{-d /2r}\right)\) neurons and fixed number of layers.

Abstract

In this work, we examine the approximation capabilities of deep neural networks utilizing the Rectified Quadratic Unit (ReQU) activation function, defined as \(\max(0,x)^2\), for approximating Hölder-regular functions with respect to the uniform norm. We constructively prove that deep neural networks with ReQU activation can approximate any function within the \(R\)-ball of \(r\)-Hölder-regular functions (\(\mathcal{H}^{r, R}([-1,1]^d)\)) up to any accuracy \(ε\) with at most \(\mathcal{O}\left(ε^{-d /2r}\right)\) neurons and fixed number of layers. This result highlights that the effectiveness of the approximation depends significantly on the smoothness of the target function and the characteristics of the ReQU activation function. Our proof is based on approximating local Taylor expansions with deep ReQU neural networks, demonstrating their ability to capture the behavior of Hölder-regular functions effectively. Furthermore, the results can be straightforwardly generalized to any Rectified Power Unit (RePU) activation function of the form \(\max(0,x)^p\) for \(p \geq 2\), indicating the broader applicability of our findings within this family of activations.

Uniform Approximation with Quadratic Neural Networks

TL;DR

It is constructively proved that deep neural networks with ReQU activation can approximate any function within the -ball of -Holder-regular functions up to any accuracy up to any accuracy with at most \(\mathcal{O}\left(\epsilon^{-d /2r}\right)\) neurons and fixed number of layers.

Abstract

In this work, we examine the approximation capabilities of deep neural networks utilizing the Rectified Quadratic Unit (ReQU) activation function, defined as \(\max(0,x)^2\), for approximating Hölder-regular functions with respect to the uniform norm. We constructively prove that deep neural networks with ReQU activation can approximate any function within the -ball of -Hölder-regular functions (\(\mathcal{H}^{r, R}([-1,1]^d)\)) up to any accuracy with at most \(\mathcal{O}\left(ε^{-d /2r}\right)\) neurons and fixed number of layers. This result highlights that the effectiveness of the approximation depends significantly on the smoothness of the target function and the characteristics of the ReQU activation function. Our proof is based on approximating local Taylor expansions with deep ReQU neural networks, demonstrating their ability to capture the behavior of Hölder-regular functions effectively. Furthermore, the results can be straightforwardly generalized to any Rectified Power Unit (RePU) activation function of the form \(\max(0,x)^p\) for , indicating the broader applicability of our findings within this family of activations.
Paper Structure (14 sections, 13 theorems, 132 equations, 2 figures, 1 table)

This paper contains 14 sections, 13 theorems, 132 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Prior and Related Work
  3. Contributions
  4. Notation
  5. Outline
  6. Preliminaries
  7. Functions of Hölder smoothness
  8. Mathematical definitions of neural networks
  9. Approximation error of smooth functions by deep ReQU neural network
  10. Proofs for Section \ref{['se3']}
  11. Proof of Lemma \ref{['le5']}
  12. Proof of Lemma \ref{['le8']}
  13. Proof of Lemma \ref{['le9']}
  14. ReQU network approximation of the square root

Key Result

Theorem 6

Let $r, R>0$, $f\in \mathcal{H}^{r, R}(\mathbb{R}^d)$ and $M \in \mathbb{N}$ such that $M>\left(\frac{cRd^{r/2}}{\epsilon}\right)^{1/2r}$, for any $\epsilon \in (0, 1)$ and $c>0$ in eq:taylor_error . Then there exists a ReQU neural network $\Phi_f \in \mathtt{N}_{\rho_2}(L(\Phi_f), N(\Phi_f))$, sati where

Figures (2)

  • Figure 1: $\bold{C^L}$ is the bottom left corner of the square $[-1, 1]^2$.
  • Figure 2: $2^2$ partitions in two dimensions.

Theorems & Definitions (20)

  • Definition 1
  • Definition 2
  • Remark 3
  • Definition 4
  • Definition 5
  • Theorem 6
  • Corollary 7: Univariate Case for the Approximation Rate
  • Remark 8: Lower Bound on the Error Rate
  • Lemma 9
  • Lemma 10
  • ...and 10 more