Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fractal and Regular Geometry of Deep Neural Networks

Simmaco Di Lillo, Domenico Marinucci, Michele Salvi, Stefano Vigogna

TL;DR

The paper classifies the geometric behavior of random neural networks at initialization by CRI, showing a fractal class for CRI in $(0,1)$ where excursion-set boundaries become increasingly fractal with depth, and a Kac–Rice class for CRI in $(1,2]$ where boundary-volume expectations are finite and depend on a single spectral parameter, $\\kappa'(1)$. It derives precise dimension results for fractal networks, and exact exponential or constant scaling laws for regular networks using the Kac–Rice formula, with a key result that CRI>1 implies a.s. $C^1$ smoothness enabling the latter analysis. The study connects the angular power spectrum decay to CRI and depth, providing a rigorous link between activation function regularity, depth, and the resulting excursion geometry on the sphere. Numerical experiments validate the theoretical dichotomy and illustrate depth-driven fractal vs. finite-volume behavior across activation types, underscoring the practical relevance for understanding deep-network initialization.

Abstract

We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for activations which are not very regular (e.g., the Heaviside step function), the boundary volumes exhibit fractal behavior, with their Hausdorff dimension monotonically increasing with the depth. On the other hand, for activations which are more regular (e.g., ReLU, logistic and $\tanh$), as the depth increases, the expected boundary volumes can either converge to zero, remain constant or diverge exponentially, depending on a single spectral parameter which can be easily computed. Our theoretical results are confirmed in some numerical experiments based on Monte Carlo simulations.

Fractal and Regular Geometry of Deep Neural Networks

TL;DR

The paper classifies the geometric behavior of random neural networks at initialization by CRI, showing a fractal class for CRI in where excursion-set boundaries become increasingly fractal with depth, and a Kac–Rice class for CRI in where boundary-volume expectations are finite and depend on a single spectral parameter, . It derives precise dimension results for fractal networks, and exact exponential or constant scaling laws for regular networks using the Kac–Rice formula, with a key result that CRI>1 implies a.s. smoothness enabling the latter analysis. The study connects the angular power spectrum decay to CRI and depth, providing a rigorous link between activation function regularity, depth, and the resulting excursion geometry on the sphere. Numerical experiments validate the theoretical dichotomy and illustrate depth-driven fractal vs. finite-volume behavior across activation types, underscoring the practical relevance for understanding deep-network initialization.

Abstract

We study the geometric properties of random neural networks by investigating the boundary volumes of their excursion sets for different activation functions, as the depth increases. More specifically, we show that, for activations which are not very regular (e.g., the Heaviside step function), the boundary volumes exhibit fractal behavior, with their Hausdorff dimension monotonically increasing with the depth. On the other hand, for activations which are more regular (e.g., ReLU, logistic and ), as the depth increases, the expected boundary volumes can either converge to zero, remain constant or diverge exponentially, depending on a single spectral parameter which can be easily computed. Our theoretical results are confirmed in some numerical experiments based on Monte Carlo simulations.

Paper Structure

This paper contains 25 sections, 22 theorems, 204 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background and definitions
  3. Spherical random fields
  4. Random neural networks
  5. Main results
  6. Spectral index and activation functions
  7. The fractal class
  8. The Kac-Rice class
  9. Proof of \ref{['teo_fractal']}: fractal class
  10. Hausdorff dimension and measure
  11. Technical tools
  12. The CRI of deep neural networks
  13. Proof of \ref{['grafoqqq']}: Hausdorff dimension of the graph
  14. Proof of \ref{['almost']}: Hausdorff dimension of the excursion set
  15. Proof of \ref{['th0']}: Kac-Rice class implies $C^1$
  16. ...and 10 more sections

Key Result

Proposition 3.4

Let $\alpha\neq 2$ be the spectral index of an isotropic Gaussian random field $T$. Then the CRI of $T$ is equal to $\min(2, \alpha/2)$.

Figures (3)

  • Figure 1: Nodal length of the excursion set for random neural networks with ReLU (a) and Heaviside (b) activation functions. As expected, in the fractal class (b) the length grows as the number of pixel increases.
  • Figure 2: Length of the boundary of the excursion set for Gaussian activation functions with different values of $a$ and threshold $u$, as a function of the depth $L$. The dashed lines represent the theoretical rate of growth (i.e. $\kappa'(1)^{L/2}$). The resolution of the maps is $0.11$ deg.
  • Figure 3: Percentage of the variance of $T_L$ explained by the first $1536$ frequencies in the case $a=9$. These values are obtained using the Gauss-Legendre quadrature to compute the angular power spectrum.

Theorems & Definitions (44)

  • Definition 3.1: Covariance Regularity Index
  • Definition 3.2: Spectral Index
  • Remark 3.3
  • Proposition 3.4
  • Definition 3.5: Fractal and Kac-Rice class
  • Remark 3.6
  • Remark 3.7: Some examples
  • Remark 3.8
  • Theorem 3.9
  • Remark 3.10
  • ...and 34 more