Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Random Vector Functional Link Networks for Function Approximation on Manifolds

Deanna Needell, Aaron A. Nelson, Rayan Saab, Palina Salanevich, Olov Schavemaker

TL;DR

This work addresses the slow training of gradient-based neural nets by rigorously analyzing Random Vector Functional Link (RVFL) networks, where input-to-hidden weights are random and output weights are learned linearly. It provides a corrected universal-approximation theorem with an asymptotic $O\left(1/\sqrt{n}\right)$ rate and a non-asymptotic, high-probability error bound, then extends these results to functions defined on smooth compact manifolds, replacing ambient-dimension dependence with intrinsic dimension $d$. The manifold extension leverages atlas-based reconstruction and local RVFLs on coordinate charts, with proofs supported by concentration bounds for Monte Carlo integration and a GMRA-based numerical demonstration. The findings offer theoretical guarantees for RVFL architectures on manifolds, highlighting potential for efficient, structure-exploiting function approximation in high-dimensional settings, albeit with room for improving dimension dependence and robustness analysis.

Abstract

The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this paper, we begin to fill this theoretical gap. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error decaying asymptotically like $O(1/\sqrt{n})$ for the number $n$ of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided $n$ is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Random Vector Functional Link Networks for Function Approximation on Manifolds

TL;DR

This work addresses the slow training of gradient-based neural nets by rigorously analyzing Random Vector Functional Link (RVFL) networks, where input-to-hidden weights are random and output weights are learned linearly. It provides a corrected universal-approximation theorem with an asymptotic rate and a non-asymptotic, high-probability error bound, then extends these results to functions defined on smooth compact manifolds, replacing ambient-dimension dependence with intrinsic dimension . The manifold extension leverages atlas-based reconstruction and local RVFLs on coordinate charts, with proofs supported by concentration bounds for Monte Carlo integration and a GMRA-based numerical demonstration. The findings offer theoretical guarantees for RVFL architectures on manifolds, highlighting potential for efficient, structure-exploiting function approximation in high-dimensional settings, albeit with room for improving dimension dependence and robustness analysis.

Abstract

The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this paper, we begin to fill this theoretical gap. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error decaying asymptotically like for the number of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments.

Paper Structure

This paper contains 23 sections, 16 theorems, 147 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation
  3. Main results
  4. Organization
  5. Materials and Methods
  6. A concentration bound for classic Monte-Carlo integration
  7. Smooth, compact manifolds in Euclidean space
  8. Results
  9. Proof of Theorem \ref{['thm: IP95_short']}
  10. Proof of Theorem \ref{['thm: IP95_short']} when $\rho\in L^1(\mathbb{R})\cap L^\infty(\mathbb{R})$
  11. Proof of Theorem \ref{['thm: IP95_short']} when $\rho' \in L^1(\mathbb{R})\cap L^\infty(\mathbb{R})$
  12. Proof of Theorem \ref{['thm: probabilistic Igelnik-Pao_short']}
  13. Results on submanifolds of Euclidean space
  14. Adapting RVFL networks to $d$-manifolds
  15. Main results on $d$-manifolds
  16. ...and 8 more sections

Key Result

Theorem 1

Let $f\in C_c(\mathbb{R}^N)$ with $K:=\mathrm{supp}(f)$ and fix any activation function $\rho$, such that either $\rho\in L^1(\mathbb{R})\cap L^\infty(\mathbb{R})$ with $\int_{\mathbb{R}}\rho(z)\mathrm{d}z=1$ or $\rho$ is differentiable with $\rho'\in L^1(\mathbb{R})\cap L^\infty(\mathbb{R})$ and $\ satisfies as $n\to\infty.$

Figures (1)

  • Figure 1: Log-scale plot of average relative error for Algorithm \ref{['alg: RVFL main']} as a function of the number of nodes $n$ in each RVFL network. Black (cross), blue (circle), and red (square) lines correspond to GMRA refinement levels $j=12$, $j=9$, and $j=6$ (resp.). For each $j$, we fix $\alpha_j=2$ and vary $\Omega_j=10,15$ (solid and dashed lines, resp.). Reconstruction error decays as a function of $n$ until reaching a floor due to error in the GMRA approximation of $\mathcal{M}$. The code used to obtain these numerical results is available upon direct request sent to the corresponding author.

Theorems & Definitions (36)

  • Theorem 1: igelnik1995stochastic
  • Remark 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • Definition 1
  • Lemma 2
  • Theorem 5: igelnik1995stochastic
  • proof
  • ...and 26 more