Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions

Christian Kuehn, Sara-Viola Kuntz

TL;DR

This work analyzes the geometric structure of the input–output maps of both finite-depth multilayer perceptrons (MLPs) and neural ODEs through Morse theory, establishing a architecture-dependent classification of critical points. It proves that non-augmented architectures have no critical points, while augmented and bottleneck configurations generically yield non-degenerate critical points (i.e., Morse maps) except for measure-zero degenerate cases; for neural ODEs, degenerate architectures force degenerate critical points. A constructive normal-form theorem shows any non-full-rank MLP is equivalent to a smaller full-rank architecture, clarifying the practical geometry of expressivity. The paper also links these geometric properties to universal embedding and approximation capabilities, proving that augmented neural ODEs admit universal embedding into $C^1(\mathcal{X},\mathbb{R})$ and, under expressive vector fields, universal approximation for parameterized augmented ODEs, while non-augmented/degenerate variants do not. Together, these results provide a rigorous bridge between network architecture, the geometry of the functions they implement, and their representational limits.

Abstract

Besides classical feed-forward neural networks such as multilayer perceptrons, also neural ordinary differential equations (neural ODEs) have gained particular interest in recent years. Neural ODEs can be interpreted as an infinite depth limit of feed-forward or residual neural networks. We study the input-output dynamics of finite and infinite depth neural networks with scalar output. In the finite depth case, the input is a state associated with a finite number of nodes, which maps under multiple non-linear transformations to the state of one output node. In analogy, a neural ODE maps an affine linear transformation of the input to an affine linear transformation of its time-$T$ map. We show that, depending on the specific structure of the network, the input-output map has different properties regarding the existence and regularity of critical points. These properties can be characterized via Morse functions, which are scalar functions where every critical point is non-degenerate. We prove that critical points cannot exist if the dimension of the hidden layer is monotonically decreasing or the dimension of the phase space is smaller than or equal to the input dimension. In the case that critical points exist, we classify their regularity depending on the specific architecture of the network. We show that except for a Lebesgue measure zero set in the weight space, each critical point is non-degenerate if for finite depth neural networks the underlying graph has no bottleneck, and if for neural ODEs, the affine linear transformations used have full rank. For each type of architecture, the proven properties are comparable in the finite and infinite depth cases. The established theorems allow us to formulate results on universal embedding and universal approximation, i.e., on the exact and approximate representation of maps by neural networks and neural ODEs.

Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions

TL;DR

This work analyzes the geometric structure of the input–output maps of both finite-depth multilayer perceptrons (MLPs) and neural ODEs through Morse theory, establishing a architecture-dependent classification of critical points. It proves that non-augmented architectures have no critical points, while augmented and bottleneck configurations generically yield non-degenerate critical points (i.e., Morse maps) except for measure-zero degenerate cases; for neural ODEs, degenerate architectures force degenerate critical points. A constructive normal-form theorem shows any non-full-rank MLP is equivalent to a smaller full-rank architecture, clarifying the practical geometry of expressivity. The paper also links these geometric properties to universal embedding and approximation capabilities, proving that augmented neural ODEs admit universal embedding into and, under expressive vector fields, universal approximation for parameterized augmented ODEs, while non-augmented/degenerate variants do not. Together, these results provide a rigorous bridge between network architecture, the geometry of the functions they implement, and their representational limits.

Abstract

Besides classical feed-forward neural networks such as multilayer perceptrons, also neural ordinary differential equations (neural ODEs) have gained particular interest in recent years. Neural ODEs can be interpreted as an infinite depth limit of feed-forward or residual neural networks. We study the input-output dynamics of finite and infinite depth neural networks with scalar output. In the finite depth case, the input is a state associated with a finite number of nodes, which maps under multiple non-linear transformations to the state of one output node. In analogy, a neural ODE maps an affine linear transformation of the input to an affine linear transformation of its time- map. We show that, depending on the specific structure of the network, the input-output map has different properties regarding the existence and regularity of critical points. These properties can be characterized via Morse functions, which are scalar functions where every critical point is non-degenerate. We prove that critical points cannot exist if the dimension of the hidden layer is monotonically decreasing or the dimension of the phase space is smaller than or equal to the input dimension. In the case that critical points exist, we classify their regularity depending on the specific architecture of the network. We show that except for a Lebesgue measure zero set in the weight space, each critical point is non-degenerate if for finite depth neural networks the underlying graph has no bottleneck, and if for neural ODEs, the affine linear transformations used have full rank. For each type of architecture, the proven properties are comparable in the finite and infinite depth cases. The established theorems allow us to formulate results on universal embedding and universal approximation, i.e., on the exact and approximate representation of maps by neural networks and neural ODEs.
Paper Structure (20 sections, 39 theorems, 177 equations, 10 figures, 1 table)

This paper contains 20 sections, 39 theorems, 177 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Overview and Results
  3. Classification of Critical Points
  4. Universal Embedding and Universal Approximation
  5. Multilayer Perceptrons
  6. Special Architectures
  7. Equivalent Neural Network Architectures
  8. Existence of Critical Points
  9. Regularity of Critical Points
  10. Analysis of Bottleneck Architectures
  11. Neural ODEs
  12. Special Architectures
  13. Existence of Critical Points
  14. Regularity of Critical Points
  15. General and Parameterized Neural ODEs
  16. ...and 5 more sections

Key Result

Theorem 2.2

Let $\mathcal{X} \subset \mathbb{R}^n$ be open and bounded. For $k \in \mathbb{N}_0$, the vector space endowed with the $C^k$-norm $\left\lVert\psi\right\rVert_{C^k(\mkern 1.5mu\overline{\mkern-1.5mu\mathcal{X}\mkern-1.5mu}\mkern 1.5mu,\mathbb{R})} \coloneqq \sum_{\left\lvert s\right\rvert \leq k} \left\lVert\partial^s \psi\right\rVert_\infty$ is a Banach space. Hereby $\mkern 1.5mu\overline{\mke

Figures (10)

  • Figure 1: Comparison of the main results for generic multilayer perceptrons and neural ODEs.
  • Figure 3.1: Structure of an MLP with update rule \ref{['eq:nn_updaterule']} in standard notation $(l\in \{0,\ldots,L\})$ and in common notation $(j \in \{0,\ldots,2L\})$ for main and intermediate layers, visualized in dark and light gray respectively.
  • Figure 3.2: The three different types of architectures non-augmented, augmented, and bottleneck, for classical feed-forward neural networks.
  • Figure 3.4: Equivalent neural network architectures of Example \ref{['ex:nn_normalform']} to illustrate Theorem \ref{['th:nn_fullrank_equivalent']}\ref{['th:nn_fullrank_equivalent_a']}.
  • Figure 3.5: Equivalent neural network architectures of Example \ref{['ex:nn_classes']} to illustrate Theorem \ref{['th:nn_classes_coordinatechange']}.
  • ...and 5 more figures

Theorems & Definitions (101)

  • Definition 2.1: Morse Function Hirsch1976Morse1934
  • Theorem 2.2: Kuehn2023
  • Definition 2.3
  • Definition 2.4: Universal Approximation Kratsios2021
  • Definition 2.5: Universal Embedding
  • Theorem 2.6
  • proof
  • Corollary 2.7
  • proof
  • Remark 2.8
  • ...and 91 more