Topological obstruction to the training of shallow ReLU neural networks

TL;DR

This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow and analytically compute the number of connected components, finding that the non-connectedness results in a topological obstruction, which can make the global optimum unreachable.

Abstract

Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings. This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments.

Figures (5)

• Figure 1: a. Depiction of the two group actions acting on the space of the network's parameters: the neuron rescaling of \ref{['eq:rescaling_action_neuron']} (top) and the neuron permutation of \ref{['eq:neuron_permutation']} (bottom). b. Depiction of the geometry of the parameter space induced by the rescaling invariance of ReLU networks. The dotted lines denote the orbits $T(\theta)$ while the solid lines represent the invariant sets $\mathcal{H}(c)$ associated with $\theta$ and the one associated with its rescaled version $\theta'$. Notice how the gradient of the loss $g(\theta)$ is tangent to $\mathcal{H}(c)$ and orthogonal to $T(\theta)$.
• Figure 2: a. The invariant hyperquadric $\mathcal{Q}(c_k)$ of a neuron with two inputs ($d=2$) and one output ($e=1$) in the cases where $c_k<0$ (left) and $c_k>0$ (right). b. Depiction of the invariant set $\mathcal{H}(c)$ in the case where $l_- = 2$ so that there are $2^{l_-} = 4$ connected components. $C_{\pm\mp}$ denotes the connected component such that $s = (\pm 1,\mp 1)$. The blue lines separate the different effective components of $\mathcal{H}(c)$.
• Figure 3: Visualization of the experimental setup described in \ref{['section:experiment']}. a. The small 2-layer neural network architecture considered. b. The hidden neurons' parameter spaces, together with the invariant hyperquadrics associated with hidden neurons 1 (left) and 2 (right), for an initialization with topological obstruction (top) and without it (bottom). The colored curves represent the gradient descent trajectories from initialization $\theta_k(0)$ up to $t_* = 500$ optimization steps. c. The loss curves for the bad (obstructed) and good initializations.
• Figure 4: Left. Average test BCE loss of a two-layer ReLU neural network trained on the breast cancer dataset over 100 different initializations for each pair ($l$,$l_+$), $l = 2,\dots,9$ and $l_+\leq l$, of numbers of hidden neurons and non-pathological neurons. Right. the y-axis displays the percentage of non-pathological neurons.
• Figure 5: Probability of the topological obstruction as a function of the number of input $d$ and hidden $l$ neurons, when the initial weights are sampled with Xavier normal (left) and Kaiming normal (right) initialization schemes.

Theorems & Definitions (27)

• Proposition 1
• Definition 1: Invariant set
• Lemma 1
• Proposition 2
• Theorem 1
• Corollary 1
• proof
• Corollary 2
• Proposition 3
• Definition 2: Effective component