Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Function and derivative approximation by shallow neural networks

Yuanyuan Li, Shuai Lu

TL;DR

This work analyzes function and derivative approximation for unknown targets on the unit cube $\Omega=(0,1)^d$ using shallow neural networks under a Tikhonov regularization framework. It introduces three network norms—the (extended) Barron norm, the variation norm, and the Radon-BV seminorm—and derives their interrelations and connections to Sobolev spaces, highlighting how the dimension $d$ affects embedding constants and error bounds. The authors establish approximation properties for each norm and provide rigorous $H^m(\Omega)$ error bounds for the regularized estimators, showing dimension-dependent rates that govern the accuracy of both function and derivative recovery. The results offer a unified view of regularization strategies for shallow nets and shed light on the dimensionality challenges, suggesting pathways to extend these insights to deep networks and broader inverse problems.

Abstract

We investigate a Tikhonov regularization scheme specifically tailored for shallow neural networks within the context of solving a classic inverse problem: approximating an unknown function and its derivatives within a unit cubic domain based on noisy measurements. The proposed Tikhonov regularization scheme incorporates a penalty term that takes three distinct yet intricately related network (semi)norms: the extended Barron norm, the variation norm, and the Radon-BV seminorm. These choices of the penalty term are contingent upon the specific architecture of the neural network being utilized. We establish the connection between various network norms and particularly trace the dependence of the dimensionality index, aiming to deepen our understanding of how these norms interplay with each other. We revisit the universality of function approximation through various norms, establish rigorous error-bound analysis for the Tikhonov regularization scheme, and explicitly elucidate the dependency of the dimensionality index, providing a clearer understanding of how the dimensionality affects the approximation performance and how one designs a neural network with diverse approximating tasks.

Function and derivative approximation by shallow neural networks

TL;DR

This work analyzes function and derivative approximation for unknown targets on the unit cube using shallow neural networks under a Tikhonov regularization framework. It introduces three network norms—the (extended) Barron norm, the variation norm, and the Radon-BV seminorm—and derives their interrelations and connections to Sobolev spaces, highlighting how the dimension affects embedding constants and error bounds. The authors establish approximation properties for each norm and provide rigorous error bounds for the regularized estimators, showing dimension-dependent rates that govern the accuracy of both function and derivative recovery. The results offer a unified view of regularization strategies for shallow nets and shed light on the dimensionality challenges, suggesting pathways to extend these insights to deep networks and broader inverse problems.

Abstract

We investigate a Tikhonov regularization scheme specifically tailored for shallow neural networks within the context of solving a classic inverse problem: approximating an unknown function and its derivatives within a unit cubic domain based on noisy measurements. The proposed Tikhonov regularization scheme incorporates a penalty term that takes three distinct yet intricately related network (semi)norms: the extended Barron norm, the variation norm, and the Radon-BV seminorm. These choices of the penalty term are contingent upon the specific architecture of the neural network being utilized. We establish the connection between various network norms and particularly trace the dependence of the dimensionality index, aiming to deepen our understanding of how these norms interplay with each other. We revisit the universality of function approximation through various norms, establish rigorous error-bound analysis for the Tikhonov regularization scheme, and explicitly elucidate the dependency of the dimensionality index, providing a clearer understanding of how the dimensionality affects the approximation performance and how one designs a neural network with diverse approximating tasks.
Paper Structure (16 sections, 13 theorems, 106 equations)

This paper contains 16 sections, 13 theorems, 106 equations.

Table of Contents

  1. Introduction
  2. Overview of the shallow neural network norms
  3. (Extended) Barron norm
  4. Variation norm
  5. Radon-BV seminorm
  6. Relationships among different norms and spaces
  7. Relationships among different network norms
  8. Relationship to the Sobolev norm
  9. Summary of the relationship among different norms
  10. Revisit of the approximation property
  11. Regularization and error bound analysis
  12. Regularization with the extended Barron norm penalty
  13. Regularization with the variation norm penalty
  14. Regularization with the Radon-BV seminorm penalty
  15. Discussion of the above error bounds
  16. ...and 1 more sections

Key Result

Proposition 2.4

\newlabelppt:RBV0 Let $f$ be a function satisfying the following representation Then where $\mu^{+}(w,b) := \frac{1}{2} \left(\mu(w,b) + \mu(-w,-b)\right)$ and $\mu^{-}(w,b) := \frac{1}{2} \left(\mu(w,b) - \mu(-w,-b)\right)$.

Theorems & Definitions (31)

  • Definition 2.1: Extended Barron spaces LLMP23
  • Definition 2.2: Variation spaces of $\mathrm{RePU}$ dictionary SX23
  • Definition 2.3: Radon-BV spaces on $\mathbb{R}^d$ PN21
  • Proposition 2.4
  • Proof 1
  • Definition 2.5: Radon-BV spaces on $\Omega$ PN23
  • Proposition 3.1
  • Proof 2
  • Lemma 3.2
  • Proof 3
  • ...and 21 more