Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Minimal Control Family of Dynamical Systems for Universal Approximation

Yifei Duan, Yongqiang Cai

TL;DR

The paper establishes that a minimal control family consisting of all affine maps plus a single nonlinear function (ReLU) suffices to uniformly approximate orientation-preserving diffeomorphisms on any compact domain, thereby achieving the universal approximation property through flow-map compositions. It provides a rigorous framework linking neural-network-style function composition to control-system flow maps, leveraging split-step methods and neural-ODE insights to construct flow-discretization-flow pipelines that realize the UAP. The authors prove several results including: (i) span-density of the control family implies $C$-UAP for $\mathrm{Diff}_0(\mathbb{R}^d)$; (ii) nonlinear affine-invariant families with Lipschitz nonlinearities achieve $C$-UAP; (iii) the simple $\mathcal{F}_1$ (affines plus ReLU) yields $C$-UAP for $d\ge2$ and its symmetric version $\mathcal{F}_2$ yields $C$-UAP for all $d$. They discuss extensions to $L^p$-UAP and $C$-UAP for broader continuous functions, the role of Lie-bracket generation, and the notion of a minimal control family, with implications for the theoretical understanding of flow-based models in machine learning.

Abstract

The universal approximation property (UAP) holds a fundamental position in deep learning, as it provides a theoretical foundation for the expressive power of neural networks. It is widely recognized that a composition of linear and nonlinear functions, such as the rectified linear unit (ReLU) activation function, can approximate continuous functions on compact domains. In this paper, we extend this efficacy to a scenario containing dynamical systems with controls. We prove that the control family $\mathcal{F}_1$ containing all affine maps and the nonlinear ReLU map is sufficient for generating flow maps that can approximate orientation-preserving (OP) diffeomorphisms on any compact domain. Since $\mathcal{F}_1$ contains only one nonlinear function and the UAP does not hold if we remove the nonlinear function, we call $\mathcal{F}_1$ a minimal control family for the UAP. On this basis, several mild sufficient conditions, such as affine invariance, are established for the control family and discussed. Our results reveal an underlying connection between the approximation power of neural networks and control systems and could provide theoretical guidance for examining the approximation power of flow-based models.

A Minimal Control Family of Dynamical Systems for Universal Approximation

TL;DR

The paper establishes that a minimal control family consisting of all affine maps plus a single nonlinear function (ReLU) suffices to uniformly approximate orientation-preserving diffeomorphisms on any compact domain, thereby achieving the universal approximation property through flow-map compositions. It provides a rigorous framework linking neural-network-style function composition to control-system flow maps, leveraging split-step methods and neural-ODE insights to construct flow-discretization-flow pipelines that realize the UAP. The authors prove several results including: (i) span-density of the control family implies -UAP for ; (ii) nonlinear affine-invariant families with Lipschitz nonlinearities achieve -UAP; (iii) the simple (affines plus ReLU) yields -UAP for and its symmetric version yields -UAP for all . They discuss extensions to -UAP and -UAP for broader continuous functions, the role of Lie-bracket generation, and the notion of a minimal control family, with implications for the theoretical understanding of flow-based models in machine learning.

Abstract

The universal approximation property (UAP) holds a fundamental position in deep learning, as it provides a theoretical foundation for the expressive power of neural networks. It is widely recognized that a composition of linear and nonlinear functions, such as the rectified linear unit (ReLU) activation function, can approximate continuous functions on compact domains. In this paper, we extend this efficacy to a scenario containing dynamical systems with controls. We prove that the control family containing all affine maps and the nonlinear ReLU map is sufficient for generating flow maps that can approximate orientation-preserving (OP) diffeomorphisms on any compact domain. Since contains only one nonlinear function and the UAP does not hold if we remove the nonlinear function, we call a minimal control family for the UAP. On this basis, several mild sufficient conditions, such as affine invariance, are established for the control family and discussed. Our results reveal an underlying connection between the approximation power of neural networks and control systems and could provide theoretical guidance for examining the approximation power of flow-based models.
Paper Structure (25 sections, 19 theorems, 67 equations, 1 table)

This paper contains 25 sections, 19 theorems, 67 equations, 1 table.

Table of Contents

  1. Introduction
  2. Notations and main results
  3. Universal approximation property
  4. Control families and diffeomorphisms
  5. Main results
  6. Theorem proofs
  7. Candidate flow maps
  8. Linear flow maps
  9. ReLU flow maps
  10. Two-piecewise linear maps
  11. Error estimation and splitting methods
  12. Proof of Theorem \ref{['th:UAP_span_F_to_HF']}
  13. Proof of Theorem \ref{['th:UAP_affine']} (the first part)
  14. Proof of Theorem \ref{['th:UAP_ReLU_family']}
  15. Proof of Theorem \ref{['th:UAP_affine']} (the second part)
  16. ...and 10 more sections

Key Result

Proposition 1

Let the map $T = F_n \circ ... \circ F_1$ be a composition of $n$ continuous functions $F_i$ defined on open domains $D_i$, and let $\mathcal{F}$ be a continuous function class that can uniformly approximate each $F_i$ on any compact domain $\mathcal{K}_i \subset D_i$ (under the uniform norm). Then,

Theorems & Definitions (36)

  • Definition 1: Universal approximation property, UAP
  • Proposition 1
  • Proposition 2
  • Definition 2: Control family
  • Proposition 3
  • Theorem 1
  • Definition 3
  • Theorem 2
  • Theorem 3
  • Remark 1
  • ...and 26 more