Bases of Lebesgue spaces formed by neural networks

TL;DR

Addresses how to build Lebesgue-space bases from a ReLU-friendly univariate basis by comparing outer tensor-product and inner-product multivariate constructions. It provides direct inner-product calculations and spectral analyses for the univariate system, proves Schauder-basis properties in $L_q$ for $1<q<\infty$, and establishes dimension-free Schauder bases via an inner-product structure ${\mathcal{R}}_n$ for all dimensions, along with improved Riesz constants in the Hilbert setting. It also clarifies the dimensional limits of tensor-product approaches (valid for $n\le 3$) and shows that inner-product-induced bases extend to all dimensions, offering a robust mathematical framework for neural-network-friendly bases in Lebesgue spaces. The results have potential implications for high-dimensional approximation using ReLU-implementable basis functions and provide precise spectral and basis-properties that underpin stable representations in $L_q$ spaces.

Abstract

The seminal work of Daubechies, DeVore, Foucart, Hanin, and Petrova introduced in 2022 a sequence of univariate piece-wise linear functions, which resemble the classical Fourier basis and which, at the same time, can be easily reproduced by artificial neural networks with ReLU activation function. We give an alternative way how to calculate the inner products of functions from this system and discuss the spectral properties of the Gram matrix generated by this system. The univariate system was later generalized to the multivariate setting by two of the authors of this work. Instead of the usual tensor product construction, this generalization relied on the inner products inside of the argument of the univariate sequence. It turned out that such a system forms a Riesz basis of $L_2(0,1)^n$ for every $n\ge 1$ with Riesz constants independent of $n$. In this work, we investigate the properties of these new sequences of functions in $L_q(0,1)^n$ for $q\not =2.$ First, we show that the univariate system is a Schauder basis in $L_q(0,1)$ for every $1<q<\infty$. By a general argument, it follows that the tensor products of this system also form a Schauder basis in $L_q(0,1)^n$ for every $n\ge 2$ and $1<q<\infty.$ The same fact can also be shown by measuring the distance of the tensor product system to the classical multivariate Fourier basis, but - surprisingly - this argument only works for $n\le 3$. If, on the other hand, we replace the outer tensor products by inner products directly in the argument of the univariate system, the same approach is applicable for an arbitrary dimension $n\in{\mathbb N}.$

Figures (3)

• Figure 1: The plot of $\mathcal{C}$, $\mathcal{S}$, $\mathcal{C}_2$ and $\mathcal{S}_2$.
• Figure 2: Hat function $S(t)$ and scaled version $S_2(t)$
• Figure 3: Illustration of the signs of $f$ in case $n=2$

Theorems & Definitions (32)

• Definition 1.1
• Lemma 2.1: SV
• proof
• Theorem 2.2
• proof
• Remark 2.3
• Proposition 2.4
• proof
• Lemma 2.5
• proof