Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal approximation property of ODENet and ResNet with a single activation function

Masato Kimura, Kazunori Matsui, Yosuke Mizuno

TL;DR

It is shown that such an ODENet and ResNet with a restricted vector field can uniformly approximate ODENet with a general vector field.

Abstract

We study a universal approximation property of ODENet and ResNet. The ODENet is a map from an initial value to the final value of an ODE system in a finite interval. It is considered a mathematical model of a ResNet-type deep learning system. We consider dynamical systems with vector fields given by a single composition of the activation function and an affine mapping, which is the most common choice of the ODENet or ResNet vector field in actual machine learning systems. We show that such an ODENet and ResNet with a restricted vector field can uniformly approximate ODENet with a general vector field.

Universal approximation property of ODENet and ResNet with a single activation function

TL;DR

It is shown that such an ODENet and ResNet with a restricted vector field can uniformly approximate ODENet with a general vector field.

Abstract

We study a universal approximation property of ODENet and ResNet. The ODENet is a map from an initial value to the final value of an ODE system in a finite interval. It is considered a mathematical model of a ResNet-type deep learning system. We consider dynamical systems with vector fields given by a single composition of the activation function and an affine mapping, which is the most common choice of the ODENet or ResNet vector field in actual machine learning systems. We show that such an ODENet and ResNet with a restricted vector field can uniformly approximate ODENet with a general vector field.

Paper Structure

This paper contains 11 sections, 12 theorems, 114 equations.

Table of Contents

  1. Introduction
  2. ODENet and universal approximation property
  3. Main theorems
  4. Preliminary results
  5. Proofs of the main theorems
  6. Proof of $\|S_f(t)-S_{f_L}(t)\|_{C^0(D;\mathbb{R}^N)}<\frac{\epsilon}{3}$
  7. Proof of $\|S_{f_L}(t)-S_{h_L}(t)\|_{C^0(D;\mathbb{R}^N)}<\frac{\epsilon}{3}$
  8. Proof of $\|S_{h_L}(t)-S_{h}(t)\|_{C^0(D;\mathbb{R}^N)}<\frac{\epsilon}{3}$
  9. Proof Summary of Theorem \ref{['main_th']}
  10. Proof of Theorem \ref{['main_th_2']}
  11. Proof of Theorem \ref{['thm:piccinini']}

Key Result

Theorem 3.1

It holds that i.e., for given $F\in\overline{{\mathcal{S}}({\mathcal{F}})}^{C^0(D;\mathbb{R}^N)}$ and $\epsilon>0$, there exists $h\in{\mathcal{H}}$ such that

Theorems & Definitions (27)

  • Definition 2.1: ODENet
  • Definition 2.2: ResNet
  • Definition 2.3: Universal approximation property for the activation function $\sigma$
  • Theorem 3.1: UAP for ODENet
  • Theorem 3.2
  • Remark 3.3
  • Remark 3.4
  • Proposition 3.5: hzh
  • proof
  • Theorem 3.6
  • ...and 17 more