Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory

TL;DR

This work reframes deep residual networks as nonlinear control systems and proves universal approximation properties using nonlinear control theory. By requiring activations (or derivatives) to satisfy a quadratic differential equation, the authors establish ensemble controllability of ResNets and leverage monotonicity to extend finite-sample controllability to uniform approximation on compact sets, achieving a width bound of $2n+1$ (with $n$ input dimensions) for $\mathbb{R}^n\to\mathbb{R}^n$ mappings. The results rely on a Lie-algebraic analysis of the ensemble system and an embedding-based scheme to connect analytic monotone homotopies to practical network architectures, with two-valued weight forms enabling simple realizations. The framework also hints at deterministic generalization bounds and suggests future work on safety and robustness when placing networks in closed-loop control contexts.

Abstract

In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property to be sufficient for an adequately deep neural network with $n+1$ neurons per layer to approximate arbitrarily well, on a compact set and with respect to the supremum norm, any continuous function from $\mathbb{R}^n$ to $\mathbb{R}^n$. We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control systems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is to identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.

Figures (1)

• Figure 1: A schematic picture of the two problems under study. The objective in Problem \ref{['problem1']} is to find a control input $(s,W,b)$ that takes all the sample points in $E_{\textup{samples}}$ (shown by red dots) to their corresponding points in $f(E)$ (shown by blue dots); some of these assignments are shown by black dashed arrows. In Problem \ref{['problem2']} we are additionally concerned with the quality of the assignments of the points outside $E_{\textup{samples}}$, shown by green dashed arrow, as we aim to guarantee approximation in the uniform sense.

Theorems & Definitions (19)

• Definition 3.1
• Remark 3.2
• Definition 4.1
• Theorem 4.2
• proof
• Corollary 4.3
• proof
• Theorem 5.1
• Corollary 5.2
• Lemma 5.3