Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Schauder Bases for $C[0, 1]$ Using ReLU, Softplus and Two Sigmoidal Functions

Anand Ganesh, Babhrubahan Bose, Anand Rajagopalan

TL;DR

The paper constructs four Schauder bases for $C[0,1]$ using fixed activation functions: a ReLU-based basis, a Softplus-based basis, and two sigmoidal variants. It establishes explicit Schauder expansions with bounded coefficient functionals and proves an $O(1/n)$ worst-case approximation bound for univariate functions using the ReLU basis. It further shows a negative result: finite linear combinations of ReLU functions cannot represent general multivariate functions, highlighting the necessity of deeper architectures for high-dimensional approximation. The work situates these bases within the broader context of universal approximation and provides potential implications for neural network initialization and understanding of approximation power, while leaving open questions about smooth bases and higher-dimensional constructions.

Abstract

We construct four Schauder bases for the space $C[0,1]$, one using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an $O(\frac{1}{n})$ approximation bound based on our ReLU basis, and a negative result on constructing multivariate functions using finite combinations of ReLU functions.

Schauder Bases for $C[0, 1]$ Using ReLU, Softplus and Two Sigmoidal Functions

TL;DR

The paper constructs four Schauder bases for using fixed activation functions: a ReLU-based basis, a Softplus-based basis, and two sigmoidal variants. It establishes explicit Schauder expansions with bounded coefficient functionals and proves an worst-case approximation bound for univariate functions using the ReLU basis. It further shows a negative result: finite linear combinations of ReLU functions cannot represent general multivariate functions, highlighting the necessity of deeper architectures for high-dimensional approximation. The work situates these bases within the broader context of universal approximation and provides potential implications for neural network initialization and understanding of approximation power, while leaving open questions about smooth bases and higher-dimensional constructions.

Abstract

We construct four Schauder bases for the space , one using ReLU functions, another using Softplus functions, and two more using sigmoidal versions of the ReLU and Softplus functions. This establishes the existence of a basis using these functions for the first time, and improves on the universal approximation property associated with them. We also show an approximation bound based on our ReLU basis, and a negative result on constructing multivariate functions using finite combinations of ReLU functions.

Paper Structure

This paper contains 7 sections, 10 theorems, 27 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. ReLU Basis
  4. Softplus Basis
  5. Applications
  6. Conclusion
  7. Function Plots

Key Result

Theorem 1

The basis functions $\chi_{[0, 1]}(x)$, $s_1(x) = x$, $r(2^n x - k)$ and $r(2^n x - (k + \frac{1}{2}))$ form a Schauder basis for $C[0, 1]$. In particular, borrowing $\alpha_{n, k}$ from the Schauder basis expansion equation eq:schauder, and setting $\alpha_{n, -1} = 0$, we have where the coefficient functionals $\alpha_0$, $\alpha_1$, $(\alpha_{n, k} + \alpha_{n, k-1})$ and $-2\alpha_{n, k}$ are

Figures (4)

  • Figure 1: ReLU function plots
  • Figure 2: First Differences $d_{n,k}(x)$
  • Figure 3: Second Differences $t_{n,k}(x) = s_{n,k}(x)$
  • Figure 4: (a) describes the counter example $g_{n, k} + 0.5$, (b) describes a perturbed Schauder built using Softplus functions $p_a(x)$ with $a=10$, and (c) shows the pyramidal function used in Theorem \ref{['negative_result_finite']}.

Theorems & Definitions (17)

  • Theorem 1
  • Lemma 1
  • proof
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Lemma 2
  • ...and 7 more