Neural Networks for Singular Perturbations
Joost A. A. Opschoor, Christoph Schwab, Christos Xenophontos
TL;DR
This work addresses uniform, exponential expressivity bounds for deep neural networks approximating solution families of singularly perturbed elliptic BVPs on $(-1,1)$ with analytic data. It develops parallel, architecture-specific analyses for strict ReLU nets, spiking nets, and tanh/sigmoid nets, showing robust exponential convergence in Sobolev norms that are uniform in the perturbation parameter $\varepsilon$. Central contributions include constructive ReLU and spiking NN results with explicit depth/size bounds, a tanh NN framework that yields accelerated, $\varepsilon$-robust rates by emulating boundary-layer structures and analytic components (via Chebyshev polynomials), and detailed calculus of NN operations to enable composition of complex emulation networks. The findings imply that deep NNs with tanh or sigmoid activations can resolve exponential boundary-layer features uniformly in length-scale without augmenting the feature space, with potential impact on multiscale PDE approximation and neuromorphic implementations. Overall, the paper provides a rigorous, architecture-aware pathway to achieving uniformly accurate DNN representations of singular perturbation solutions in Sobolev spaces.
Abstract
We prove deep neural network (DNN for short) expressivity rate bounds for solution sets of a model class of singularly perturbed, elliptic two-point boundary value problems, in Sobolev norms, on the bounded interval $(-1,1)$. We assume that the given source term and reaction coefficient are analytic in $[-1,1]$. We establish expression rate bounds in Sobolev norms in terms of the NN size which are uniform with respect to the singular perturbation parameter for several classes of DNN architectures. In particular, ReLU NNs, spiking NNs, and $\tanh$- and sigmoid-activated NNs. The latter activations can represent ``exponential boundary layer solution features'' explicitly, in the last hidden layer of the DNN, i.e. in a shallow subnetwork, and afford improved robust expression rate bounds in terms of the NN size. We prove that all DNN architectures allow robust exponential solution expression in so-called `energy' as well as in `balanced' Sobolev norms, for analytic input data.