Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation Capabilities of Feedforward Neural Networks with GELU Activations

Konstantin Yakovlev, Nikita Puchkin

TL;DR

The paper develops a constructive theory for approximating multivariate functions and all prescribed derivatives using deep GELU neural networks on unbounded domains. By combining localized Taylor-polynomial techniques with clipping and partition-of-unity constructions, it obtains explicit Sobolev-error bounds that hold uniformly as the domain grows, and provides detailed network-architecture bounds (depth, width, sparsity, and weight magnitudes) for a suite of elementary operations, monomials, exponentiation, and division. Key contributions include global boundedness of higher-order derivatives via input clipping, and scalable, provable approximations of polynomials and multivariate functions with Sobolev-norm accuracy across expanding domains. The results extend prior domain-restricted analyses and yield practical, quantifiable guarantees for building GELU-based approximators with controlled smoothness and stability for applications in generative modeling and physics-informed learning.

Abstract

We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher-order derivatives of the resulting approximators remain globally bounded.

Approximation Capabilities of Feedforward Neural Networks with GELU Activations

TL;DR

The paper develops a constructive theory for approximating multivariate functions and all prescribed derivatives using deep GELU neural networks on unbounded domains. By combining localized Taylor-polynomial techniques with clipping and partition-of-unity constructions, it obtains explicit Sobolev-error bounds that hold uniformly as the domain grows, and provides detailed network-architecture bounds (depth, width, sparsity, and weight magnitudes) for a suite of elementary operations, monomials, exponentiation, and division. Key contributions include global boundedness of higher-order derivatives via input clipping, and scalable, provable approximations of polynomials and multivariate functions with Sobolev-norm accuracy across expanding domains. The results extend prior domain-restricted analyses and yield practical, quantifiable guarantees for building GELU-based approximators with controlled smoothness and stability for applications in generative modeling and physics-informed learning.

Abstract

We derive an approximation error bound that holds simultaneously for a function and all its derivatives up to any prescribed order. The bounds apply to elementary functions, including multivariate polynomials, the exponential function, and the reciprocal function, and are obtained using feedforward neural networks with the Gaussian Error Linear Unit (GELU) activation. In addition, we report the network size, weight magnitudes, and behavior at infinity. Our analysis begins with a constructive approximation of multiplication, where we prove the simultaneous validity of error bounds over domains of increasing size for a given approximator. Leveraging this result, we obtain approximation guarantees for division and the exponential function, ensuring that all higher-order derivatives of the resulting approximators remain globally bounded.
Paper Structure (13 sections, 20 theorems, 332 equations)

This paper contains 13 sections, 20 theorems, 332 equations.

Table of Contents

  1. Introduction
  2. Preliminaries and notations
  3. Main results
  4. Approximation of elementary operations
  5. Approximation of monomials
  6. Approximation of the exponent and the division
  7. Deferred proofs
  8. Proof of Lemma \ref{['lem:id_deep_gelu_approx']}
  9. Proof of Corollary \ref{['co:multi_approx_gelu']}
  10. Proof of Corollary \ref{['co:monom_approx_gelu']}
  11. Proof of Lemma \ref{['lem:div_approx_naive']}
  12. Proof of Lemma \ref{['lem:div_gelu_approx']}
  13. Auxiliary results

Key Result

Lemma 3.1

Let $m \in \mathbb N$ and let $\mathrm{id}(x) = x$. Then, for any $\varepsilon \in (0, 1)$ there exists $\varphi_{id} \in \mathsf{NN}(L, W, S, B)$ satisfying Furthermore, $L = 2$, $\|W\|_\infty = 1$, $S = 3$, and $\log B \lesssim \log(1 / \varepsilon) + \log m$.

Theorems & Definitions (34)

  • Definition 2.1: Sobolev space
  • Lemma 3.1: approximation of identity operation
  • proof
  • Lemma 3.2: approximation of identity operation with multiple layers
  • Lemma 3.3: Approximation of Heaviside step function
  • proof
  • Lemma 3.4: partition of unity approximation
  • proof
  • Lemma 3.5: approximation of clipping operation
  • proof
  • ...and 24 more