Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Geometry of ReLU Networks through the ReLU Transition Graph

Sahil Rajesh Dhayalkar

TL;DR

The paper introduces the ReLU Transition Graph (RTG), a graph whose nodes are activation-induced linear regions and edges connect regions differing by a single ReLU flip, to study the global geometry of ReLU networks. It establishes a theoretical framework linking RTG size, connectivity, entropy, and diameter to expressivity and generalization, including a VC-dimension bound via RTG diameter and a sparsity-driven compression theorem. The authors prove key results (Theorem 1, Theorem 2, Theorem 3, Theorem 4) and validate them through unified experiments on fully connected ReLU MLPs with synthetic 2D data, demonstrating region counting scaling, adjacency correctness, connectivity, entropy behavior, and practical pruning with bounded error. The RTG framework offers a graph-theoretic lens for understanding ReLU networks, enabling principled compression, regularization, and potential extensions to convolutional, residual, and attention-based architectures with real data.

Abstract

We develop a novel theoretical framework for analyzing ReLU neural networks through the lens of a combinatorial object we term the ReLU Transition Graph (RTG). In this graph, each node corresponds to a linear region induced by the network's activation patterns, and edges connect regions that differ by a single neuron flip. Building on this structure, we derive a suite of new theoretical results connecting RTG geometry to expressivity, generalization, and robustness. Our contributions include tight combinatorial bounds on RTG size and diameter, a proof of RTG connectivity, and graph-theoretic interpretations of VC-dimension. We also relate entropy and average degree of the RTG to generalization error. Each theoretical result is rigorously validated via carefully controlled experiments across varied network depths, widths, and data regimes. This work provides the first unified treatment of ReLU network structure via graph theory and opens new avenues for compression, regularization, and complexity control rooted in RTG analysis.

The Geometry of ReLU Networks through the ReLU Transition Graph

TL;DR

The paper introduces the ReLU Transition Graph (RTG), a graph whose nodes are activation-induced linear regions and edges connect regions differing by a single ReLU flip, to study the global geometry of ReLU networks. It establishes a theoretical framework linking RTG size, connectivity, entropy, and diameter to expressivity and generalization, including a VC-dimension bound via RTG diameter and a sparsity-driven compression theorem. The authors prove key results (Theorem 1, Theorem 2, Theorem 3, Theorem 4) and validate them through unified experiments on fully connected ReLU MLPs with synthetic 2D data, demonstrating region counting scaling, adjacency correctness, connectivity, entropy behavior, and practical pruning with bounded error. The RTG framework offers a graph-theoretic lens for understanding ReLU networks, enabling principled compression, regularization, and potential extensions to convolutional, residual, and attention-based architectures with real data.

Abstract

We develop a novel theoretical framework for analyzing ReLU neural networks through the lens of a combinatorial object we term the ReLU Transition Graph (RTG). In this graph, each node corresponds to a linear region induced by the network's activation patterns, and edges connect regions that differ by a single neuron flip. Building on this structure, we derive a suite of new theoretical results connecting RTG geometry to expressivity, generalization, and robustness. Our contributions include tight combinatorial bounds on RTG size and diameter, a proof of RTG connectivity, and graph-theoretic interpretations of VC-dimension. We also relate entropy and average degree of the RTG to generalization error. Each theoretical result is rigorously validated via carefully controlled experiments across varied network depths, widths, and data regimes. This work provides the first unified treatment of ReLU network structure via graph theory and opens new avenues for compression, regularization, and complexity control rooted in RTG analysis.
Paper Structure (33 sections, 27 equations, 4 figures, 1 table)

This paper contains 33 sections, 27 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Expressivity via Region Counting:
  4. Connectivity and Topology of ReLU Regions:
  5. Generalization and VC-Dimension:
  6. Graph-Theoretic Perspectives on Neural Networks:
  7. Sparsity, Pruning, and Functional Compression:
  8. Spectral and Entropy-Based Measures:
  9. Preliminaries and Definitions
  10. Definition 1: Activation Pattern
  11. Definition 2: Linear Region
  12. Definition 3: ReLU Transition Graph (RTG)
  13. Definition 4: RTG Entropy
  14. Theoretical Framework
  15. Theorem 1: RTG Size Bound
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Empirical RTG node counts vs. theoretical upper bound for various network depths $L$ and widths $n$. All empirical values remain below the bound, validating Theorem \ref{['theorem_1']}.
  • Figure 2: Validation of Lemma \ref{['lemma_2']}. Each point corresponds to a network configuration. All lie above the diagonal, showing that $H(G) \geq \log(d_{\text{avg}} + 1)$.
  • Figure 3: VC-dimension proxy vs. RTG diameter across $(L, n)$ configurations. Dashed line indicates $y = x$. For all points, the VC-dimension proxy is $<=$ the RTG diametere, validating Theorem \ref{['theorem_3']}.
  • Figure 4: Validation of Theorem \ref{['theorem_4']}. Each point shows approximation error vs. compression rate after pruning 50% of RTG nodes (by degree). All configurations maintain bounded error.