Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Universal Approximation Theorem to Tropical Geometry of Multi-Layer Perceptrons

Yi-Shan Chu, Yueh-Cheng Kuo

TL;DR

The paper tackles the problem that classical universal approximation results certify existence but not construction of networks with prescribed geometry. It develops a geometry-aware initializer for purely sigmoidal MLPs by compiling a target planar region into a finite-sum network of sigmoid half-space gates, guided by tropical boundary reasoning. The method yields decision boundaries that align with prescribed shapes at initialization, with training serving mainly for refinement; it also extends to arbitrary planar unions via finite ball covers and to higher dimensions through ball-to-polytope constructions. Empirical studies in 2D demonstrate strong initialization performance on disks and Swiss-roll targets, highlighting the practical potential of shaping inductive bias directly through geometry. Overall, the work bridges the classical UAT framework with tropical boundary reasoning, providing an interpretable, shape-driven initialization strategy for smooth MLPs.

Abstract

We revisit the Universal Approximation Theorem(UAT) through the lens of the tropical geometry of neural networks and introduce a constructive, geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs). Tropical geometry shows that Rectified Linear Unit (ReLU) networks admit decision functions with a combinatorial structure often described as a tropical rational, namely a difference of tropical polynomials. Focusing on planar binary classification, we design purely sigmoidal MLPs that adhere to the finite-sum format of UAT: a finite linear combination of shifted and scaled sigmoids of affine functions. The resulting models yield decision boundaries that already align with prescribed shapes at initialization and can be refined by standard training if desired. This provides a practical bridge between the tropical perspective and smooth MLPs, enabling interpretable, shape-driven initialization without resorting to ReLU architectures. We focus on the construction and empirical demonstrations in two dimensions; theoretical analysis and higher-dimensional extensions are left for future work.

From Universal Approximation Theorem to Tropical Geometry of Multi-Layer Perceptrons

TL;DR

The paper tackles the problem that classical universal approximation results certify existence but not construction of networks with prescribed geometry. It develops a geometry-aware initializer for purely sigmoidal MLPs by compiling a target planar region into a finite-sum network of sigmoid half-space gates, guided by tropical boundary reasoning. The method yields decision boundaries that align with prescribed shapes at initialization, with training serving mainly for refinement; it also extends to arbitrary planar unions via finite ball covers and to higher dimensions through ball-to-polytope constructions. Empirical studies in 2D demonstrate strong initialization performance on disks and Swiss-roll targets, highlighting the practical potential of shaping inductive bias directly through geometry. Overall, the work bridges the classical UAT framework with tropical boundary reasoning, providing an interpretable, shape-driven initialization strategy for smooth MLPs.

Abstract

We revisit the Universal Approximation Theorem(UAT) through the lens of the tropical geometry of neural networks and introduce a constructive, geometry-aware initialization for sigmoidal multi-layer perceptrons (MLPs). Tropical geometry shows that Rectified Linear Unit (ReLU) networks admit decision functions with a combinatorial structure often described as a tropical rational, namely a difference of tropical polynomials. Focusing on planar binary classification, we design purely sigmoidal MLPs that adhere to the finite-sum format of UAT: a finite linear combination of shifted and scaled sigmoids of affine functions. The resulting models yield decision boundaries that already align with prescribed shapes at initialization and can be refined by standard training if desired. This provides a practical bridge between the tropical perspective and smooth MLPs, enabling interpretable, shape-driven initialization without resorting to ReLU architectures. We focus on the construction and empirical demonstrations in two dimensions; theoretical analysis and higher-dimensional extensions are left for future work.
Paper Structure (34 sections, 14 theorems, 83 equations, 8 figures, 1 table, 3 algorithms)

This paper contains 34 sections, 14 theorems, 83 equations, 8 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work.
  3. Universal Approximation for Sigmoidal MLPs
  4. Universal approximation (UAT).
  5. Tropical Geometry
  6. A one–dimensional warm-up: least–squares construction
  7. Model and design matrix.
  8. Closed-form weights.
  9. Binary classification outputs.
  10. Choice of centers and sharpness.
  11. Summary.
  12. Planar Convex Regions and Unions
  13. Tropical preface.
  14. Tropical interpretation.
  15. Polyhedral approximation and Gates.
  16. ...and 19 more sections

Key Result

Theorem 2.2

Let $\sigma$ be sigmoidal in the sense of Definition def:sigmoidal. Then, for any compact $K\subset\mathbb{R}^d$, the set of finite sums eq:finite-sum is dense in $C(K)$ with respect to the sup norm, i.e., for every $f\in C(K)$ and every $\varepsilon>0$, there exist $N$ and parameters $\{\alpha_j,w_

Figures (8)

  • Figure 1: The tropical line $x\oplus y\oplus 0$: three rays at the origin.
  • Figure 2: Tropical hypersurfaces $\mathcal{T}(f)$ and subdivision of its Newton polygons $\delta(f)$. Top: $\mathcal{T}(f)$—unions of 1, 2, and 3 tropical lines (apices $(0,0)$, $(1,-1)$, $(-1,1)$). Bottom: $\delta(f)$—Newton triangles of sizes $1,2,3$. Red dashed lines are straight outward primitive normals (left $(-1,0)$, bottom $(0,-1)$, diagonal $(1,1)$), each drawn as a single segment that runs from an interior anchor point, across the edge, and extends outside the polygon.
  • Figure 3: One–dimensional function fitting with the least–squares initializer. Top row: varying sigmoid sharpness $k$. Middle row: varying center placements $p_j$ with sigmoid(k=20). Bottom row: different shapes reconstructed using algorithm \ref{['alg:1d-ls']} without training.
  • Figure 4: How Farthest Point Sampling choose centers
  • Figure 5: Distributions used in the three cases, positives are colored in orange, while negatives are in gray.
  • ...and 3 more figures

Theorems & Definitions (42)

  • Definition 2.1: Sigmoidal activation; Cybenko1989
  • Theorem 2.2: Cybenko's UAT; Cybenko1989
  • Theorem 2.3: Nonpolynomial criterion Leshno1993
  • Definition 3.1: Tropical semiring (max--plus)
  • Proposition 3.2: Basic properties of tropical semiring
  • Remark 3.3: Min--plus convention
  • Definition 3.4
  • Definition 3.5: Monomials and polynomials
  • Remark 3.6: Tropical powers
  • Proposition 3.7: Convex piecewise-linear structure
  • ...and 32 more