A Theoretical Study of Neural Network Expressive Power via Manifold Topology

TL;DR

This study investigates network expressive power in terms of the latent data manifold by integrating both topological and geometric facets of the data manifold and presents a size upper bound of ReLU neural networks.

Abstract

A prevalent assumption regarding real-world data is that it lies on or close to a low-dimensional manifold. When deploying a neural network on data manifolds, the required size, i.e., the number of neurons of the network, heavily depends on the intricacy of the underlying latent manifold. While significant advancements have been made in understanding the geometric attributes of manifolds, it's essential to recognize that topology, too, is a fundamental characteristic of manifolds. In this study, we investigate network expressive power in terms of the latent data manifold. Integrating both topological and geometric facets of the data manifold, we present a size upper bound of ReLU neural networks.

Figures (6)

• Figure 1: Illustration of Betti numbers and reach. (a) A 2-manifold embedding in $\mathbb{R}^3$ with $\beta_0=1,\beta_1=0$. (b) A 2-manifold embedded in $\mathbb{R}^3$ with $\beta_0=1,\beta_1=3$. (c) A 1-manifold with large reach. (d) A 1-manifold with small reach, which is the radius of the dashed circle.
• Figure 2: An illustration of the 1-thickened 1-manifold in 2D space. The top row shows a manifold that has the same homotopy type as a closed interval $I^1$, while the bottom row shows a manifold that is homotopy equivalent to a circle $S^1$. The 1-thickened 1-manifold family contains all manifolds that can be obtained through the disjoint union and connected sum of these manifolds.
• Figure 3: Construction of the network $g$. The network first learns a low-dimensional embedding and then performs classification in the embedding space. This paradigm mirrors the typical operation of deep networks. While the diagram illustrates only the process for the manifold of the positive class, the procedure for ${\mathcal{M}}_0$ mirrors this operation identically.
• Figure A.1: Illustration of connected sum.
• Figure A.2: Network construction.

Theorems & Definitions (32)

• Definition 1: Thickened $1$-Manifold Family
• Definition 2: Topological Complexity
• Definition 3: Reach and Condition Number
• Definition 4: Adapted from arora2018understanding
• Definition 5: Approximation Error
• Lemma 1: Topological Representative
• Proposition 1: Approximating a $\mathbb{R}^d$ Ball, adapted from Theorem 2 in Safran2016DepthWidthTI
• Proposition 2: Approximating a Solid Torus
• Theorem 1: Complexity Arising from Topology
• Proposition 3: Theorem 3.1 in Niyogi2008Homology