Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyperplane Arrangements and Fixed Points in Iterated PWL Neural Networks

Hans-Peter Beise

TL;DR

This work analyzes fixed points in multi-layer neural networks with piecewise-linear activations by mapping activation regions to hyperplane arrangements. Using a gp/ph general-position modulo parallel hyperplane framework, it derives an upper bound on the number of fixed points that factor across layers as a product of per-layer region counts, showing exponential growth with the number of layers, and proves this growth is asymptotically optimal via a saw-tooth construction. It also provides a sharper bound for one-hidden-layer networks with hard tanh by bounding stable fixed points with a region-count term r_d(n,1), and demonstrates substantial numerical improvements over naive Zaslavsky-based counts. The results deepen our understanding of the complexity of fixed points in PWL networks and pave the way for extensions to smooth activations and broader architectures.

Abstract

We leverage the framework of hyperplane arrangements to analyze potential regions of (stable) fixed points. We provide an upper bound on the number of fixed points for multi-layer neural networks equipped with piecewise linear (PWL) activation functions with arbitrary many linear pieces. The theoretical optimality of the exponential growth in the number of layers of the latter bound is shown. Specifically, we also derive a sharper upper bound on the number of stable fixed points for one-hidden-layer networks with hard tanh activation.

Hyperplane Arrangements and Fixed Points in Iterated PWL Neural Networks

TL;DR

This work analyzes fixed points in multi-layer neural networks with piecewise-linear activations by mapping activation regions to hyperplane arrangements. Using a gp/ph general-position modulo parallel hyperplane framework, it derives an upper bound on the number of fixed points that factor across layers as a product of per-layer region counts, showing exponential growth with the number of layers, and proves this growth is asymptotically optimal via a saw-tooth construction. It also provides a sharper bound for one-hidden-layer networks with hard tanh by bounding stable fixed points with a region-count term r_d(n,1), and demonstrates substantial numerical improvements over naive Zaslavsky-based counts. The results deepen our understanding of the complexity of fixed points in PWL networks and pave the way for extensions to smooth activations and broader architectures.

Abstract

We leverage the framework of hyperplane arrangements to analyze potential regions of (stable) fixed points. We provide an upper bound on the number of fixed points for multi-layer neural networks equipped with piecewise linear (PWL) activation functions with arbitrary many linear pieces. The theoretical optimality of the exponential growth in the number of layers of the latter bound is shown. Specifically, we also derive a sharper upper bound on the number of stable fixed points for one-hidden-layer networks with hard tanh activation.
Paper Structure (8 sections, 14 theorems, 57 equations, 1 figure)

This paper contains 8 sections, 14 theorems, 57 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Prelimaries
  3. Arrangements of Parallel and Non-Parallel Hyperplanes
  4. Linear Regions and Fixed Points of Multi-Layer Neural Networks
  5. Regions with Stable Fixed Points
  6. Numerical experiments
  7. Conclusion
  8. Auxiliary Results and Proofs

Key Result

Proposition 3.2

For $n,k\in\mathbb{N}$, the number of regions for every $\mathcal{H}\in \Lambda(n,k)$ is given by

Figures (1)

  • Figure 1: Left: Plot of $\log_{10}(\gamma(n,k,d))$, as defined in (\ref{['def_gamma_ratio']}), with respect to the number of neurons $n$ for $\vert\mathcal{A}(\phi)\vert-1=k=2,5,10$ and $d=15,25$. Right: Plot of $\log_{10}(\eta(n,d))$, as defined in (\ref{['def_eta_ratio']}), with respect to the number of neurons $n$ and for $d=15,20,30$.

Theorems & Definitions (35)

  • Definition 3.1
  • Proposition 3.2
  • proof
  • Definition 3.3
  • Proposition 3.4
  • proof
  • Definition 4.1
  • Definition 4.2
  • Lemma 4.3
  • Theorem 4.4
  • ...and 25 more