Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

Christian Kuehn, Sara-Viola Kuntz, Tobias Wöhrer

Abstract

We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.

Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect

Abstract

We analyze the universal approximation constraints of narrow Residual Neural Networks (ResNets) both theoretically and numerically. For deep neural networks without input space augmentation, a central constraint is the inability to represent critical points of the input-output map. We prove that this has global consequences for target function approximations and show that the manifestation of this defect is typically a shift of the critical point to infinity, which we call the ``tunnel effect'' in the context of classification tasks. While ResNets offer greater expressivity than standard multilayer perceptrons (MLPs), their capability strongly depends on the signal ratio between the skip and residual channels. We establish quantitative approximation bounds for both the residual-dominant (close to MLP) and skip-dominant (close to neural ODE) regimes. These estimates depend explicitly on the channel ratio and uniform network weight bounds. Low-dimensional examples further provide a detailed analysis of the different ResNet regimes and how architecture-target incompatibility influences the approximation error.

Paper Structure

This paper contains 36 sections, 20 theorems, 91 equations, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Critical Points in Residual Neural Networks
  3. Universal Approximation and Critical Points
  4. Residual Neural Networks
  5. Assumptions on Non-Augmented Architectures
  6. Existence of Critical Points
  7. Global Topological Restrictions
  8. Topological Restrictions of Maps Without Critical Points
  9. One-Dimensional Case
  10. Higher-Dimensional Case
  11. Extension to Networks in Close Proximity
  12. One-Dimensional Case
  13. Higher-Dimensional Case
  14. Case $0<\alpha \ll 1$: Close to Neural ODEs
  15. Neural ODEs
  16. ...and 21 more sections

Key Result

Theorem 2.3

Consider a set of functions $\mathcal{S}\subset C^1(\mathcal{X},\mathbb{R}^{n_\textup{out}})$, $\mathcal{X}\subset \mathbb{R}^{n_\textup{in}}$ open, wherein for every map $\Phi \in \mathcal{S}$, there exists a component $i\in \{1,\ldots,n_\textup{out}\}$, such that $\nabla_x\Phi_i(x) \neq 0$ for all

Figures (12)

  • Figure 1: The "tunnel effect" in non-augmented MLPs.
  • Figure 1: Summary of the main results regarding the relationship between ResNets, neural ODEs, and FNNs, and the existence of critical points in the parameter regimes $0<\alpha \ll 1$ and $\alpha \gg 1$.
  • Figure 2: Expressivity of the non-augmented ResNets depending on the ratio $\alpha \coloneqq \frac{\delta}{\varepsilon}$ and the upper and lower Lipschitz constants of the residual function.
  • Figure 3: Structure of the update rule of a residual neural network with skip parameter $\varepsilon$ and residual parameter $\delta$. Every layer $h_l \in \mathbb{R}^{n_\textup{hid}}$ is represented by a square.
  • Figure 4: Classification of ResNet architectures depending on the input and the hidden dimension. Every node of the neural network is represented as a circle.
  • ...and 7 more figures

Theorems & Definitions (71)

  • Definition 2.1: Universal Approximation kk2025
  • Definition 2.2: Universal Embedding kk2025
  • Theorem 2.3: Maps without Critical Points are not Universal Approximators
  • proof
  • Definition 2.4: Residual Neural Network
  • Remark 2.5
  • Definition 2.6: ResNet Classification
  • Remark 2.7
  • Remark 2.8
  • Remark 2.9
  • ...and 61 more