Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities

Marco Nurisso, Pierrick Leroy, Giovanni Petri, Francesco Vaccarino

TL;DR

The paper addresses the topology and singularities of the invariant parameter space for DAG-based ReLU networks trained with gradient flow. It formalizes conservation laws arising from ReLU homogeneity and encodes them into the invariant set $\mathcal{H}_G(c)$, defined by $\tilde{B}\theta^{2}=c$, and then derives a network-flow-based characterization of connectedness via bottleneck nodes. It further shows that singularities correspond to subnetworks disconnected from inputs/outputs and are generically unreachable under gradient flow, but can be actively promoted via a nuclear-norm regularizer to enable differentiable pruning. Empirical results on synthetic and real data support the theoretical findings and reveal that L1 regularization can induce similar pruning effects, highlighting a topology-aware path to model simplification. Overall, the work links graph topology to optimization geometry, providing principled pruning strategies that exploit invariants of the learning dynamics.

Abstract

Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.

Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities

TL;DR

The paper addresses the topology and singularities of the invariant parameter space for DAG-based ReLU networks trained with gradient flow. It formalizes conservation laws arising from ReLU homogeneity and encodes them into the invariant set , defined by , and then derives a network-flow-based characterization of connectedness via bottleneck nodes. It further shows that singularities correspond to subnetworks disconnected from inputs/outputs and are generically unreachable under gradient flow, but can be actively promoted via a nuclear-norm regularizer to enable differentiable pruning. Empirical results on synthetic and real data support the theoretical findings and reveal that L1 regularization can induce similar pruning effects, highlighting a topology-aware path to model simplification. Overall, the work links graph topology to optimization geometry, providing principled pruning strategies that exploit invariants of the learning dynamics.

Abstract

Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.
Paper Structure (42 sections, 20 theorems, 47 equations, 13 figures)

This paper contains 42 sections, 20 theorems, 47 equations, 13 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. Setup and notation
  4. DAG neural networks.
  5. Symmetries of ReLU networks.
  6. Local conservation laws under gradient flow.
  7. Invariant sets.
  8. Geometry and topology of the invariant set
  9. Non-emptiness
  10. Connectedness
  11. Numerical experiments
  12. Singularities
  13. Singularities are sub-networks.
  14. Singularities are rare.
  15. Inducing singularities.
  16. ...and 27 more sections

Key Result

Proposition 1

Let $\Tilde{B}\in\mathbb{R}^{|\Tilde{V}|\times |E|}$ be the incidence matrix of $G$ with the rows associated with input and output nodes removed; then $\llangle \theta, g(\theta) \rrangle_v = 0\ \forall v\in\Tilde{V}$ is equivalent to

Figures (13)

  • Figure 1: a. Example of a feed-forward DAG architecture $G$. b. The incidence matrix $\tilde{B}$ of $G$ with rows associated to input and output neurons removed. c. Visualization of the rescaling symmetry of ReLU neurons. d. The initialization determines the balance value $c_v = \llangle \theta, \theta \rrangle_v$ of every hidden neuron, which characterizes the shape of the invariant set (e).
  • Figure 2: Overview of connectedness.a. In- and out-bottlenecks in $G$. b. The non-emptiness of $\mathcal{H}_G(c)$ is guaranteed if every hidden neuron has input and output edges. c. Different connectedness conditions and intuitive visualizations of the associated algebraic varieties for an out-bottleneck d. Numerical experiment showcasing training dynamics in a connected and disconnected scenario for a DAG network with $3$ hidden nodes (d.1).
  • Figure 3: a. A singularity of the invariant set corresponds to a configuration where a set of neurons is cut out from input and outputs. b. Visualization of the training dynamics on an invariant set with singularities. c. Proportion of null singular values along training for shallow network with 20 hidden neurons with and without regularization. d. Test losses as a function of the number of neurons pruned. Shaded regions in c. and d. denote confidence intervals over 50 independent trainings that have converged to a low loss solution.
  • Figure 4: Visualization of the induced network flow problem $(FF_v^+)$ at a bottleneck node $v\in V_B^+$. In the right panel, we depict the internal arcs in gray, the incoming arcs in orange, the source arcs in blue, the sink arcs in red and the circulation arc in black.
  • Figure 5: (Rows) architectures (Left column) number of almost 0 singular values (threshold: $10^{-3}$) along training. (Center column) pruning neurons on trained networks using $s_k$ (multiplicative) and (Right column)$s'_k$ (maximum) scores
  • ...and 8 more figures

Theorems & Definitions (35)

  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Definition 1: Invariant set, generalization of nurisso2024topological
  • Proposition 3: Feasible balance
  • Proposition 4: de2023topology Proposition 4.7
  • Definition 2: Bottleneck neurons
  • Theorem 1
  • proof
  • ...and 25 more