Improved universal approximation with neural networks studied via affine-invariant subspaces of $L_2(\mathbb{R}^n)$

TL;DR

This work addresses universal approximation in $L_2(\\mathbb{R}^n)$ under invertible affine transformations by proving that affine-invariant closed subspaces are trivial, which hews closely to Wiener's Tauberian principles. It develops a measure-theoretic reduction to translation-invariant subspaces and uses Lebesgue density arguments to show that any nontrivial invariant subspace must be all of $L_2(\\mathbb{R}^n)$. Consequently, in dimension one, any nonzero $f  L_2(\\mathbb{R})$ yields a one-hidden-layer neural-network family that is dense in $L_2(\\mathbb{R})$, extending Wiener's Tauberian perspectives. The results extend to $L_p(\\mathbb{R})$ for $p>1$ and provide practical implications for flexible activation choices in neural networks.

Abstract

We show that there are no non-trivial closed subspaces of $L_2(\mathbb{R}^n)$ that are invariant under invertible affine transformations. We apply this result to neural networks showing that any nonzero $L_2(\mathbb{R})$ function is an adequate activation function in a one hidden layer neural network in order to approximate every function in $L_2(\mathbb{R})$ with any desired accuracy. This generalizes the universal approximation properties of neural networks in $L_2(\mathbb{R})$ related to Wiener's Tauberian Theorems. Our results extend to the spaces $L_p(\mathbb{R})$ with $p>1$.

Theorems & Definitions (17)

• Theorem 1: Neural network approximation in $L_2(\mathbb{R})$
• Theorem 2: Affine-invariant subspaces in $L_2(\mathbb{R}^n)$
• Theorem 3
• proof
• Theorem 4: Lebesgue density theorem
• proof
• Lemma 5
• proof
• Lemma 6
• proof