Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation

Joost A. A. Opschoor, Christoph Schwab

TL;DR

This work analyzes how deep ReLU networks can efficiently emulate univariate continuous, piecewise polynomial functions by exploiting Chebyshev expansions. It introduces Chebyshev-based NN emulations whose output-layer weights encode Chebyshev coefficients, computed from Clenshaw–Curtis samples via IFFT, yielding explicit network-size bounds and improved Sobolev error estimates compared to monomial-based approaches. The authors extend emulation results across a broad spectrum of function spaces, including finite-order Sobolev, fractional Sobolev, weighted Sobolev, and weighted Gevrey classes, obtaining exponential rates for analytic functions with point singularities and establishing a Chebfun–NN interoperability framework. They also apply these techniques to univariate finite element spaces (hpFEM, spectral methods) and weighted Gevrey functions, delivering sharper depth/size dependencies on polynomial degree and enabling stable, efficient neural emulations of hp-based FE and spectral approximations. Overall, the paper advances constructive, Chebyshev-based NN emulation methodology with broad applicability to hpFEM, spectral methods, and beyond, while offering practical pathways to integrate Chebfun-like capabilities into neural-network-based numerical solvers.

Abstract

We show expression rates and stability in Sobolev norms of deep feedforward ReLU neural networks (NNs) in terms of the number of parameters defining the NN for continuous, piecewise polynomial functions, on arbitrary, finite partitions $\mathcal{T}$ of a bounded interval $(a,b)$. Novel constructions of ReLU NN surrogates encoding function approximations in terms of Chebyshev polynomial expansion coefficients are developed which require fewer neurons than previous constructions. Chebyshev coefficients can be computed easily from the values of the function in the Clenshaw--Curtis points using the inverse fast Fourier transform. Bounds on expression rates and stability are obtained that are superior to those of constructions based on ReLU NN emulations of monomials as considered in [Opschoor, Petersen and Schwab, 2020] and [Montanelli, Yang and Du, 2021]. All emulation bounds are explicit in terms of the (arbitrary) partition of the interval, the target emulation accuracy and the polynomial degree in each element of the partition. ReLU NN emulation error estimates are provided for various classes of functions and norms, commonly encountered in numerical analysis. In particular, we show exponential ReLU emulation rate bounds for analytic functions with point singularities and develop an interface between Chebfun approximations and constructive ReLU NN emulations.

Deep ReLU networks and high-order finite element methods II: Chebyshev emulation

TL;DR

This work analyzes how deep ReLU networks can efficiently emulate univariate continuous, piecewise polynomial functions by exploiting Chebyshev expansions. It introduces Chebyshev-based NN emulations whose output-layer weights encode Chebyshev coefficients, computed from Clenshaw–Curtis samples via IFFT, yielding explicit network-size bounds and improved Sobolev error estimates compared to monomial-based approaches. The authors extend emulation results across a broad spectrum of function spaces, including finite-order Sobolev, fractional Sobolev, weighted Sobolev, and weighted Gevrey classes, obtaining exponential rates for analytic functions with point singularities and establishing a Chebfun–NN interoperability framework. They also apply these techniques to univariate finite element spaces (hpFEM, spectral methods) and weighted Gevrey functions, delivering sharper depth/size dependencies on polynomial degree and enabling stable, efficient neural emulations of hp-based FE and spectral approximations. Overall, the paper advances constructive, Chebyshev-based NN emulation methodology with broad applicability to hpFEM, spectral methods, and beyond, while offering practical pathways to integrate Chebfun-like capabilities into neural-network-based numerical solvers.

Abstract

We show expression rates and stability in Sobolev norms of deep feedforward ReLU neural networks (NNs) in terms of the number of parameters defining the NN for continuous, piecewise polynomial functions, on arbitrary, finite partitions of a bounded interval . Novel constructions of ReLU NN surrogates encoding function approximations in terms of Chebyshev polynomial expansion coefficients are developed which require fewer neurons than previous constructions. Chebyshev coefficients can be computed easily from the values of the function in the Clenshaw--Curtis points using the inverse fast Fourier transform. Bounds on expression rates and stability are obtained that are superior to those of constructions based on ReLU NN emulations of monomials as considered in [Opschoor, Petersen and Schwab, 2020] and [Montanelli, Yang and Du, 2021]. All emulation bounds are explicit in terms of the (arbitrary) partition of the interval, the target emulation accuracy and the polynomial degree in each element of the partition. ReLU NN emulation error estimates are provided for various classes of functions and norms, commonly encountered in numerical analysis. In particular, we show exponential ReLU emulation rate bounds for analytic functions with point singularities and develop an interface between Chebfun approximations and constructive ReLU NN emulations.
Paper Structure (20 sections, 19 theorems, 91 equations)

This paper contains 20 sections, 19 theorems, 91 equations.

Table of Contents

  1. Quantitative improvements over previous results
  2. Chebfun
  3. Preliminaries
  4. Polynomial inverse inequalities
  5. Piecewise polynomials
  6. Weighted function spaces on $I$
  7. Finite order Sobolev spaces
  8. Finite order weighted Sobolev spaces
  9. Finite order fractional Sobolev spaces
  10. Weighted Gevrey classes
  11. Polynomial interpolation in Clenshaw--Curtis points
  12. ReLU NN approximation of univariate functions
  13. ReLU neural network calculus
  14. ReLU emulation of polynomials
  15. ReLU emulation of piecewise polynomial functions
  16. ...and 5 more sections

Key Result

Lemma 3.1

For all $k\in\mathbb{N}$ and $r\in[1,\infty]$ there exists a constant $C(k,r) > 0$ such that for all $p\in\mathbb{N}$ and $\widehat{v} \in \mathbb{P}_p([-1,1])$ For $k=1$, the constant satisfies the uniform bound $C(1,r) \leq 6e^{1+1/e}$ for all $r\in[1,\infty]$. For all $k\in\mathbb{N}$ it holds that $C(k,r)\leq C(1,r)^k$. In addition, with respect to the $L^\infty$-norm we have the sharper const

Theorems & Definitions (31)

  • Lemma 3.1: Markov inequality
  • proof
  • Proposition 3.2
  • proof
  • Lemma 3.3
  • proof
  • Definition 4.1: Neural Network, Realization $\mathrm R$ OPS2020
  • Proposition 4.2: Parallelization $\mathrm P$, PV2018, OPS2020
  • Proposition 4.3: Full parallelization ${\mathrm{FP}}$ EGJS2021, OPS2020
  • Definition 4.4: Concatenation, PV2018
  • ...and 21 more