Global universal approximation of functional input maps on weighted spaces

TL;DR

The paper develops a global universal approximation theory for neural networks operating on infinite-dimensional, weighted input spaces by extending Stone-Weierstrass to a weighted, possibly non-compact setting. It introduces functional input neural networks with additive hidden layers and demonstrates density in weighted function spaces, enabling global approximation of a broad class of maps, including non-anticipative path functionals and functionals of path signatures. It further connects these approximation results to rough path theory and Gaussian process regression with signature kernels, establishing a theoretical link to uncertainty quantification via Cameron-Martin spaces. Numerical examples illustrate the framework by learning path-dependent functionals such as running maximums and other signature-based representations, highlighting the practical relevance for stochastic analysis and finance.

Abstract

We introduce so-called functional input neural networks defined on a possibly infinite dimensional weighted space with values also in a possibly infinite dimensional output space. To this end, we use an additive family to map the input weighted space to the hidden layer, on which a non-linear scalar activation function is applied to each neuron, and finally return the output via some linear readouts. Relying on Stone-Weierstrass theorems on weighted spaces, we can prove a global universal approximation result on weighted spaces for continuous functions going beyond the usual approximation on compact sets. This then applies in particular to approximation of (non-anticipative) path space functionals via functional input neural networks. As a further application of the weighted Stone-Weierstrass theorem we prove a global universal approximation result for linear functions of the signature. We also introduce the viewpoint of Gaussian process regression in this setting and emphasize that the reproducing kernel Hilbert space of the signature kernels are Cameron-Martin spaces of certain Gaussian processes. This paves a way towards uncertainty quantification for signature kernel regression.

Figures (6)

• Figure 1: A functional input neural network $\varphi: X \rightarrow Y$ with additive family $\mathcal{H}$, activation function $\rho \in C^0(\mathbb{R})$, linear readout $\mathcal{L} \subseteq Y$, and $N = 3$ number of neurons.
• Figure 2: A deep two-hidden-layer FNN $\varphi: X \rightarrow Y$ with $(N_1,N_2) = (2,2)$.
• Figure 3: Learning performance
• Figure 4: Three samples of the test set
• Figure 6: Learning performance

Theorems & Definitions (88)

• Definition 2.1
• Remark 2.2
• Example 2.3
• Remark 2.4
• Definition 2.5
• Remark 2.6
• Lemma 2.7
• proof
• Theorem 3.1: Weierstrass, weierstrass85
• Theorem 3.2: Stone-Weierstrass on $C^0(X)$, stone48 or rudin91