Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation of RKHS Functionals by Neural Networks

Tian-Yi Zhou, Namjoon Suh, Guang Cheng, Xiaoming Huo

TL;DR

The paper proves that functionals defined on reproducing kernel Hilbert spaces (RKHS) can be universally approximated by tanh neural networks using only function value samples at a finite set of points. It provides explicit error bounds for RKHS functionals induced by Sobolev, inverse multiquadric, and Gaussian kernels, and extends the results to general RKHS functionals. The approach replaces traditional basis expansions with point-evaluation interpolants in RKHSs, enabling simple input representations for functional regression tasks. The work also demonstrates how these approximation guarantees enable neural-network-based functional regression maps, with a structured proof strategy leveraging nodal projections, power functions, and Lipschitz network approximation. Overall, it advances the theoretical foundations for neural approximation of infinite-dimensional functionals and informs functional data analysis and operator-learning methods.

Abstract

Motivated by the abundance of functional data such as time series and images, there has been a growing interest in integrating such data into neural networks and learning maps from function spaces to R (i.e., functionals). In this paper, we study the approximation of functionals on reproducing kernel Hilbert spaces (RKHS's) using neural networks. We establish the universality of the approximation of functionals on the RKHS's. Specifically, we derive explicit error bounds for those induced by inverse multiquadric, Gaussian, and Sobolev kernels. Moreover, we apply our findings to functional regression, proving that neural networks can accurately approximate the regression maps in generalized functional linear models. Existing works on functional learning require integration-type basis function expansions with a set of pre-specified basis functions. By leveraging the interpolating orthogonal projections in RKHS's, our proposed network is much simpler in that we use point evaluations to replace basis function expansions.

Approximation of RKHS Functionals by Neural Networks

TL;DR

The paper proves that functionals defined on reproducing kernel Hilbert spaces (RKHS) can be universally approximated by tanh neural networks using only function value samples at a finite set of points. It provides explicit error bounds for RKHS functionals induced by Sobolev, inverse multiquadric, and Gaussian kernels, and extends the results to general RKHS functionals. The approach replaces traditional basis expansions with point-evaluation interpolants in RKHSs, enabling simple input representations for functional regression tasks. The work also demonstrates how these approximation guarantees enable neural-network-based functional regression maps, with a structured proof strategy leveraging nodal projections, power functions, and Lipschitz network approximation. Overall, it advances the theoretical foundations for neural approximation of infinite-dimensional functionals and informs functional data analysis and operator-learning methods.

Abstract

Motivated by the abundance of functional data such as time series and images, there has been a growing interest in integrating such data into neural networks and learning maps from function spaces to R (i.e., functionals). In this paper, we study the approximation of functionals on reproducing kernel Hilbert spaces (RKHS's) using neural networks. We establish the universality of the approximation of functionals on the RKHS's. Specifically, we derive explicit error bounds for those induced by inverse multiquadric, Gaussian, and Sobolev kernels. Moreover, we apply our findings to functional regression, proving that neural networks can accurately approximate the regression maps in generalized functional linear models. Existing works on functional learning require integration-type basis function expansions with a set of pre-specified basis functions. By leveraging the interpolating orthogonal projections in RKHS's, our proposed network is much simpler in that we use point evaluations to replace basis function expansions.
Paper Structure (26 sections, 20 theorems, 156 equations)

This paper contains 26 sections, 20 theorems, 156 equations.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation
  4. Main Contributions
  5. Organization of this work
  6. Motivations
  7. Main Results I: Approximation of RKHS Functionals induced by specific kernels
  8. Approximation of Functionals on Sobolev Space $W^r_2(\mathcal{X})$
  9. Approximation of RKHS Functionals Induced by Inverse Multiquadric Kernel
  10. Approximation of RKHS Functionals Induced by Gaussian Kernel
  11. Main Results II: Approximation of general RKHS Functionals
  12. Approximate Regression Map in Functional Regression Using Neural Network
  13. Proof Sketch of Main Results
  14. Approximation of Functions via Nodal functions
  15. Estimation of the Power Function
  16. ...and 11 more sections

Key Result

Lemma 3.2

A translation-invariant kernel $K$ induced by $\phi: \mathbb R^d \to \mathbb R$ is a reproducing kernel (i.e., positive semi-definite) if its Fourier transform $\widehat{\phi}$ is real-valued and non-negative.

Theorems & Definitions (45)

  • Definition 1.1: Standard Fully-connected Network
  • Definition 1.2: Functional MLP
  • Definition 1.3: RKHS
  • Definition 3.1: Definition of translation-invariant kernel
  • Lemma 3.2
  • Proposition 3.3
  • Theorem 3.4: Approximation of Functionals on Sobolev Space
  • Remark 3.5
  • Theorem 3.6: Approximation of RKHS Functionals induced by inverse multiquadric Kernel
  • Remark 3.7
  • ...and 35 more