Universality and approximation bounds for echo state networks with random weights
Zhen Li, Yunfei Yang
TL;DR
This work investigates the capacity of echo state networks (ESNs) with randomly generated internal weights to uniformly approximate continuous causal time-invariant operators. The authors develop a sampling procedure for the internal weights that, under mild conditions on the activation $\phi$, yields high-probability universal approximation via ESNs trained only on readouts. They establish a general framework for random neural network universality and extend it to ESNs, with explicit results for ReLU activations and a reconstruction-based proof that leverages the uniform law of large numbers and Barron-type integral representations. For ReLU networks, they derive concrete approximation bounds, showing that the error decays like $\mathcal{O}(\sqrt{\log n / n})$ up to a memory-term $\mathcal{E}_F(m)$, and they discuss how to balance memory with sample size to achieve fast rates. The findings bridge theory and practice by explaining how random reservoirs can achieve universal approximation while outlining limitations and directions for future work on practical weight-sampling schemes and ESP guarantees.
Abstract
We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators.