Achieving Universal Approximation and Universal Interpolation via Nonlinearity of Control Families
Yongqiang Cai, Yifei Duan
TL;DR
This work establishes precise conditions under which generalized control families can realize universal approximation and interpolation for flow maps of orientation-preserving diffeomorphisms, connecting dynamical controllability with function-approximation capabilities. By introducing and analyzing control families ${\mathcal F}_{\text{ass}}(f)$, ${\mathcal F}_{\text{aff}}(f)$, and ${\mathcal F}_{\text{diag}}(f)$, the paper derives sufficient criteria for $C$-UAP and UIP under the uniform norm, and introduces local UIP to bridge gaps when global UIP is hard to verify. A key contribution is showing that ${\mathcal F}_{\text{ass}}(f)$ achieves UAP in various nonlinear settings (including coordinate-separable nonlinearities and certain non-coordinate cases) and that ${\mathcal F}_{\mathrm{diag}}(f)$ attains UIP under suitable nonlinearities, with a unifying result that UAP plus local UIP implies UIP. Additionally, the analysis clarifies the relationship between associated affine and affine-invariant families, demonstrating that ${\mathcal F}_{\text{aff}}(f)$ is typically more expressive, and proves UIP for ${\mathcal F}_{\text{ass}}(\mathrm{ReLU})$ via a local interpolation argument. Overall, the results articulate when nonlinear control nonlinearity suffices to approximate all orientation-preserving diffeomorphisms, with implications for neural-network-inspired discretizations of flow maps and their interpolation capabilities.
Abstract
A significant connection exists between the controllability of dynamical systems and the approximation capabilities of neural networks, where residual networks and vanilla feedforward neural networks can both be regarded as numerical discretizations of the flow maps of dynamical systems. Leveraging the expressive power of neural networks, prior works have explored various control families $\mathcal{F}$ that enable the flow maps of dynamical systems to achieve either the universal approximation property (UAP) or the universal interpolation property (UIP). For example, the control family $\mathcal{F}_\text{ass}({\mathrm{ReLU}})$, consisting of affine maps together with a specific nonlinear function ReLU, achieves UAP; while the affine-invariant nonlinear control family $\mathcal{F}_{\mathrm{aff}}(f)$ containing a nonlinear function $f$ achieves UIP. However, UAP and UIP are generally not equivalent, and thus typically need to be studied separately with different techniques. In this paper, we investigate more general control families, including $\mathcal{F}_\text{ass}(f)$ with nonlinear functions $f$ beyond ReLU, the diagonal affine-invariant family $\mathcal{F}_{\mathrm{diag}}(f)$, and UAP for orientation-preserving diffeomorphisms under the uniform norm. We show that in certain special cases, UAP and UIP are indeed equivalent, whereas in the general case, we introduce the notion of local UIP (a substantially weaker version of UIP) and prove that the combination of UAP and local UIP implies UIP. In particular, the control family $\mathcal{F}_\text{ass}({\mathrm{ReLU}})$ achieves the UIP.