Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Univariate Radial Basis Function Layers: Brain-inspired Deep Neural Layers for Low-Dimensional Inputs

Daniel Jost, Basavasagar Patil, Xavier Alameda-Pineda, Chris Reinke

TL;DR

This work introduces the Univariate-RBF (U-RBF) layer to address function-approximation for low-dimensional inputs by per-dimension Gaussian kernels with learnable centers $\mu_{d,k}$ and spreads $\sigma_{d,k}$, expanding each input $x_d$ into a high-dimensional feature set $z_{d,k}=\mathcal{G}(x_d-c_{d,k},\sigma_{d,k})$ before a readout layer. A universal function-approximation result is established when a U-RBF layer is followed by a hidden layer, supported by a corollary that ensures a one-to-one mapping from the input space to the U-RBF feature space. Empirically, U-RBF variants outperform several baselines on white-noise regression and real-world low-dimensional datasets, while performing competitively with Fourier-feature approaches and sometimes underperforming on image regression where traditional methods excel. The findings suggest U-RBF offers a robust, brain-inspired, and parameter-efficient alternative for low-dimensional regression tasks, with simpler hyperparameter requirements than Fourier-feature mappings. Overall, the approach broadens the toolbox for low-dimensional function approximation and control tasks, including potential reinforcement-learning applications mentioned in the abstract.

Abstract

Deep Neural Networks (DNNs) became the standard tool for function approximation with most of the introduced architectures being developed for high-dimensional input data. However, many real-world problems have low-dimensional inputs for which standard Multi-Layer Perceptrons (MLPs) are the default choice. An investigation into specialized architectures is missing. We propose a novel DNN layer called Univariate Radial Basis Function (U-RBF) layer as an alternative. Similar to sensory neurons in the brain, the U-RBF layer processes each individual input dimension with a population of neurons whose activations depend on different preferred input values. We verify its effectiveness compared to MLPs in low-dimensional function regressions and reinforcement learning tasks. The results show that the U-RBF is especially advantageous when the target function becomes complex and difficult to approximate.

Univariate Radial Basis Function Layers: Brain-inspired Deep Neural Layers for Low-Dimensional Inputs

TL;DR

This work introduces the Univariate-RBF (U-RBF) layer to address function-approximation for low-dimensional inputs by per-dimension Gaussian kernels with learnable centers and spreads , expanding each input into a high-dimensional feature set before a readout layer. A universal function-approximation result is established when a U-RBF layer is followed by a hidden layer, supported by a corollary that ensures a one-to-one mapping from the input space to the U-RBF feature space. Empirically, U-RBF variants outperform several baselines on white-noise regression and real-world low-dimensional datasets, while performing competitively with Fourier-feature approaches and sometimes underperforming on image regression where traditional methods excel. The findings suggest U-RBF offers a robust, brain-inspired, and parameter-efficient alternative for low-dimensional regression tasks, with simpler hyperparameter requirements than Fourier-feature mappings. Overall, the approach broadens the toolbox for low-dimensional function approximation and control tasks, including potential reinforcement-learning applications mentioned in the abstract.

Abstract

Deep Neural Networks (DNNs) became the standard tool for function approximation with most of the introduced architectures being developed for high-dimensional input data. However, many real-world problems have low-dimensional inputs for which standard Multi-Layer Perceptrons (MLPs) are the default choice. An investigation into specialized architectures is missing. We propose a novel DNN layer called Univariate Radial Basis Function (U-RBF) layer as an alternative. Similar to sensory neurons in the brain, the U-RBF layer processes each individual input dimension with a population of neurons whose activations depend on different preferred input values. We verify its effectiveness compared to MLPs in low-dimensional function regressions and reinforcement learning tasks. The results show that the U-RBF is especially advantageous when the target function becomes complex and difficult to approximate.
Paper Structure (22 sections, 3 theorems, 13 equations, 15 figures, 2 tables)

This paper contains 22 sections, 3 theorems, 13 equations, 15 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Function Approximation
  4. Fourier Feature Mapping
  5. RBF Networks
  6. Univariate Radial Basis Function Layer
  7. Universal Function Approximation
  8. Experiments
  9. White Noise Regression
  10. Frequency Sweep
  11. Input Neuron Sweep
  12. Dimension Sweep
  13. FFM Hyperparameter Sweep
  14. Image Regression
  15. Real World Dataset Regression
  16. ...and 7 more sections

Key Result

Theorem 3.1

Let $\{\bm{x}_1, \bm{x}_2, \dots, \bm{x}_N\}$ be a set of distinct points in $\mathbb{R}^D$ and let $f : \mathbb{R}^D \to \mathbb{R}^L$ be any arbitrary function. Then there is a function $g: \mathbb{R}^D \to \mathbb{R}^L$ with urbf layer with $K \ge 2$ Gaussian kernels with different kernels, such

Figures (15)

  • Figure 1: Several neurons encode a single continuous input with Gaussian-shaped activity curves that peak at different receptive values.
  • Figure 2: The urbf layer.
  • Figure 3: Sample of isotropic low-pass filtered White Noise which serves as the target function. The cut-off frequency is set to $4 Hz$.
  • Figure 4: The U-RBF outperforms competing approaches across different models for white noise regression. The term 'fixed' refers to non-learnable RBF Neurons while 'random' and 'uniform' refer to the initialization strategies.
  • Figure 5: Performance across different noise cut-off frequencies. The urbf approach shows the best performance for the lower four frequencies.
  • ...and 10 more figures

Theorems & Definitions (5)

  • Theorem 3.1
  • Definition 1.1
  • Definition 1.2
  • Theorem 1.3
  • Corollary 1.4