Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable wavelet-based neural approximation

Youngmi Hur, Hyojae Lim, Mikyoung Lim

TL;DR

This work develops a provable wavelet-frame framework for neural network approximation on spaces of homogeneous type, linking activation functions to wavelet averaging kernels to obtain $L^2$-error bounds for approximations of functions in the $\mathcal{L}_1$ class. By requiring activation functions to satisfy certain decay and symmetry properties, the authors show that the induced wavelet system forms a frame, enabling $N$-term approximations with error decaying like $(N+1)^{-1/2}$; these approximants can be realized by a $W\vec{B}$-Net with $2N$ hidden nodes. They further extend the theory to oscillatory activations and, crucially, to non-smooth activations through an $L^2$-distance control between activations, providing corollaries that quantify the trade-off between smoothness and approximation accuracy. The results broaden the design space for neural architectures by providing concrete, provable convergence guarantees for a wider class of activation functions, with practical guidance on preserving network structure while handling non-smoothness. Overall, this work strengthens the theoretical foundation for wavelet-based neural approximation and informs architecture choices for problems with oscillatory or non-smooth activation behavior.

Abstract

In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the spaces of homogeneous type, we derive sufficient conditions on activation functions to ensure that the associated neural network approximates any functions in the given space, along with an error estimate. These sufficient conditions accommodate a variety of smooth activation functions, including those that exhibit oscillatory behavior. Furthermore, by considering the $L^2$-distance between smooth and non-smooth activation functions, we establish a generalized approximation result that is applicable to non-smooth activations, with the error explicitly controlled by this distance. This provides increased flexibility in the design of network architectures.

Provable wavelet-based neural approximation

TL;DR

This work develops a provable wavelet-frame framework for neural network approximation on spaces of homogeneous type, linking activation functions to wavelet averaging kernels to obtain -error bounds for approximations of functions in the class. By requiring activation functions to satisfy certain decay and symmetry properties, the authors show that the induced wavelet system forms a frame, enabling -term approximations with error decaying like ; these approximants can be realized by a -Net with hidden nodes. They further extend the theory to oscillatory activations and, crucially, to non-smooth activations through an -distance control between activations, providing corollaries that quantify the trade-off between smoothness and approximation accuracy. The results broaden the design space for neural architectures by providing concrete, provable convergence guarantees for a wider class of activation functions, with practical guidance on preserving network structure while handling non-smoothness. Overall, this work strengthens the theoretical foundation for wavelet-based neural approximation and informs architecture choices for problems with oscillatory or non-smooth activation behavior.

Abstract

In this paper, we develop a wavelet-based theoretical framework for analyzing the universal approximation capabilities of neural networks over a wide range of activation functions. Leveraging wavelet frame theory on the spaces of homogeneous type, we derive sufficient conditions on activation functions to ensure that the associated neural network approximates any functions in the given space, along with an error estimate. These sufficient conditions accommodate a variety of smooth activation functions, including those that exhibit oscillatory behavior. Furthermore, by considering the -distance between smooth and non-smooth activation functions, we establish a generalized approximation result that is applicable to non-smooth activations, with the error explicitly controlled by this distance. This provides increased flexibility in the design of network architectures.

Paper Structure

This paper contains 11 sections, 10 theorems, 70 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Neural Networks
  4. Wavelet expansions on spaces of homogeneous type
  5. Wavelet-based neural approximation
  6. Main results
  7. Proof of Proposition \ref{['thm:averaging']}
  8. Neural networks with non-smooth activation functions
  9. Conclusion
  10. Connection between ${\vec{W}}$B-Net and W${\vec{B}}$-Net
  11. Architectural overview of related work on wavelet-based neural approximations

Key Result

Theorem 1.1

Let $\sigma:\mathbb{R}^d \to \mathbb{R}$ be a twice-differentiable function satisfying $\int_{\mathbb{R}^d} \sigma(x)dx =1$, $\sigma(-x)= \sigma(x)$, and a decaying condition eqn: activation condition for AK. Then for every $f\in\mathcal{L}_1$ and every $N\in\mathbb{N}$, there exists a parameter set of the W$\vec{B}$-Net $\Psi_{\text{W}\vec{B}}$ with the activation function $\sigma$ such that

Figures (2)

  • Figure 3.1: An architectural sketch of our network in Theorem \ref{['corollary: smooth WNN approximation']}.
  • Figure B.1: An architectural sketch of radial quadratic neural network in Frischauf:2024:QNN.

Theorems & Definitions (27)

  • Theorem 1.1
  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Theorem 2.5: Cybenko:1989:ASS
  • Definition 2.6
  • Definition 2.7
  • Definition 2.8
  • Theorem 2.9
  • ...and 17 more