Upper Approximation Bounds for Neural Oscillators

TL;DR

This work provides quantitative upper bounds for a neural oscillator (a second-order ODE followed by an MLP) used to learn causal mappings over continuous temporal functions and to emulate uniformly stable second-order dynamical systems. The authors develop a two-part theoretical framework: Theorem 1 bounds errors for causal, uniformly continuous operators via sine-transform encoding and MLP readouts, while Theorem 2 extends to uniformly asymptotically incrementally stable second-order systems with error decaying polynomially in the reciprocal of the MLP widths. The bounds are complemented by a rigorous proof strategy that leverages sine-encoding, compact-domain arguments, and Barron-class function approximation. Numerical experiments on a Bouc–Wen system validate the predicted decay rates (-1/7 and -0.5) and demonstrate practical applicability to long-horizon temporal learning in engineering contexts.

Abstract

Neural oscillators, originating from the second-order ordinary differential equations (ODEs), have demonstrated competitive performance in stably learning causal mappings between long-term sequences or continuous temporal functions. However, theoretically quantifying the capacities of their neural network architectures remains a significant challenge. In this study, the neural oscillator consisting of a second-order ODE followed by a multilayer perceptron (MLP) is considered. Its upper approximation bound for approximating causal and uniformly continuous operators between continuous temporal function spaces and that for approximating uniformly asymptotically incrementally stable second-order dynamical systems are derived. The established proof method of the approximation bound for approximating the causal continuous operators can also be directly applied to state-space models consisting of a linear time-continuous complex recurrent neural network followed by an MLP. Theoretical results reveal that the approximation error of the neural oscillator for approximating the second-order dynamical systems scales polynomially with the reciprocals of the widths of two utilized MLPs, thus mitigating the curse of parametric complexity. The decay rates of two established approximation error bounds are validated through two numerical cases. These results provide a robust theoretical foundation for the effective application of the neural oscillator in science and engineering.

Figures (2)

• Figure 1: Neural oscillator approximation errors $\tilde{\varepsilon}_{E_X,\infty}$ and $\tilde{\varepsilon}_{E_X,2}$ versus the number $H_{{\it{\Pi}}_i}$ of hidden layers in ${\it{\Pi}}_i(\cdot)$. (a) $\tilde{\varepsilon}_{E_X,\infty}$ versus $H_{{\it{\Pi}}_i}$. (b) $\tilde{\varepsilon}_{E_X,2}$ versus $H_{{\it{\Pi}}_i}$.
• Figure 2: Neural oscillator approximation errors $\tilde{\varepsilon}_{X,\infty}$ and $\tilde{\varepsilon}_{X,2}$ versus the hidden layer width $w_{{{\it{\Gamma}}}_i}$ of ${\it{\Gamma}}_i(\cdot)$. (a) $\tilde{\varepsilon}_{X,\infty}$ versus $w_{{\it{\Gamma}}_i}$. (b) $\tilde{\varepsilon}_{X,2}$ versus $w_{{\it{\Gamma}}_i}$.