Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, Johanni Brea

TL;DR

This work deciphers the loss-landscape geometry of overparameterized neural networks by exploiting permutation symmetries. It introduces expansion manifolds that connect discrete global minima into a single zero-loss manifold and characterizes symmetry-induced critical points as affine subspaces generated by replications and permutations, with precise counts G(r,m) and T(r,m). The authors derive closed-form formulas and asymptotics showing saddles dominate in mildly overparameterized regimes, while global minima dominate in vastly overparameterized regimes, with depth intensifying these effects. The results give a principled view of optimization dynamics and provide a basis for pruning strategies that exploit replicated units. Overall, the paper advances understanding of how overparameterization and symmetries shape non-convex optimization in neural networks.

Abstract

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with $ L $ layers of minimal widths $ r_1^*, \ldots, r_{L-1}^* $ reaches a zero-loss minimum at $ r_1^*! \cdots r_{L-1}^*! $ isolated points that are permutations of one another, we show that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold. For a two-layer overparameterized network of width $ r^*+ h =: m $ we explicitly describe the manifold of global minima: it consists of $ T(r^*, m) $ affine subspaces of dimension at least $ h $ that are connected to one another. For a network of width $m$, we identify the number $G(r,m)$ of affine subspaces containing only symmetry-induced critical points that are related to the critical points of a smaller network of width $r<r^*$. Via a combinatorial analysis, we derive closed-form formulas for $ T $ and $ G $ and show that the number of symmetry-induced critical subspaces dominates the number of affine subspaces forming the global minima manifold in the mildly overparameterized regime (small $ h $) and vice versa in the vastly overparameterized regime ($h \gg r^*$). Our results provide new insights into the minimization of the non-convex loss function of overparameterized neural networks.

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

TL;DR

This work deciphers the loss-landscape geometry of overparameterized neural networks by exploiting permutation symmetries. It introduces expansion manifolds that connect discrete global minima into a single zero-loss manifold and characterizes symmetry-induced critical points as affine subspaces generated by replications and permutations, with precise counts G(r,m) and T(r,m). The authors derive closed-form formulas and asymptotics showing saddles dominate in mildly overparameterized regimes, while global minima dominate in vastly overparameterized regimes, with depth intensifying these effects. The results give a principled view of optimization dynamics and provide a basis for pruning strategies that exploit replicated units. Overall, the paper advances understanding of how overparameterization and symmetries shape non-convex optimization in neural networks.

Abstract

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with layers of minimal widths reaches a zero-loss minimum at isolated points that are permutations of one another, we show that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold. For a two-layer overparameterized network of width we explicitly describe the manifold of global minima: it consists of affine subspaces of dimension at least that are connected to one another. For a network of width , we identify the number of affine subspaces containing only symmetry-induced critical points that are related to the critical points of a smaller network of width . Via a combinatorial analysis, we derive closed-form formulas for and and show that the number of symmetry-induced critical subspaces dominates the number of affine subspaces forming the global minima manifold in the mildly overparameterized regime (small ) and vice versa in the vastly overparameterized regime (). Our results provide new insights into the minimization of the non-convex loss function of overparameterized neural networks.

Paper Structure

This paper contains 22 sections, 22 theorems, 111 equations, 12 figures.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Work
  4. Symmetric Losses
  5. Foundations: Invariances in 2-Layer ANNs
  6. Overparameterized ANN Landscapes
  7. The global minima manifold
  8. Symmetry-induced critical points
  9. Width-dependent comparison of the critical subspaces and the global minima subspaces
  10. Multi-Layer ANNs
  11. Conclusion & Discussion
  12. Further Experimental Results
  13. MNIST Experiments for Two-Layer ANNs with Various Widths
  14. A Detailed Analysis of the Number of Critical Subspaces and the Number of Global Minima Subspaces
  15. Symmetric Loss Landscape Examples
  16. ...and 7 more sections

Key Result

Lemma 2.1

Let $L^m : \mathbb R^{Dm} \to \mathbb R$ be a symmetric loss on $m$ units thus a $C^1$ function and let $\pmb{\rho}: \mathbb R_{\geq0} \to \mathbb R^{Dm}$ be its gradient flow. If $\pmb{\rho} (0) \in \mathcal{H}_{i_1, \ldots, i_k }$, the gradient flow stays inside the symmetry subspace, i.e. $\pmb{\

Figures (12)

  • Figure 1: Graph of (a) a minimal network of width 4 (teacher) and (b) a mildly overparameterized student network of width 5. (c) With 50 random initializations, mildly overparameterized networks (blue) find a global minimum for only a fraction of initializations, whereas vastly overparameterized networks (red, width 45) consistently find a global minimum. (d) Graph of student network with three hidden layer learning from a teacher with widths $4-4-4$. (e) Vastly overparameterized networks (red) consistently find a global minimum whereas mildly overparameterized networks (blue) typically do not.
  • Figure 2: No gradient pointing outside of a symmetry subspace. The gradient flow of a permutation-symmetric loss $L(w_1, w_2)= \log(\frac{1}{2} ((w_1 + w_2 - 3)^2 + (w_1 w_2 - 2)^2 ) + 1)$. Red: permutation-symmetric global minima, purple: saddle, dashed line: the symmetry subspace.
  • Figure 3: Left: Parameters $\pmb{\theta}^r$ of an irreducible point in a network of $r$ neurons with $w_i \neq w_j$ for all $i \neq j$ and $a_i \neq 0$ for all $i$. Right: example of a reducible point in $\Gamma_{s}(\pmb{\theta}^r)$ in an expanded network of $m>r$ neurons. The incoming weight vector of the first neuron is replicated $k_1$ times, the second one only once, etc.
  • Figure 4: The geometry of the expansion manifold $\Theta_{r \to m}$ with $m=r+1$ and the connectivity graph of the affine subspaces. The arrangement of the subspaces is demonstrated geometrically only in (a)-(b), but their connectivity graph is shown in all three cases. Blue subspaces have one vanishing output weight, green subspaces have two identical incoming weight vectors. (a) Case of a network with two hidden neurons with parameters $(w_1, w', a_1, 0)$ that is reducible to a network with a single hidden neuron. The base subspace $\Gamma_0$ is connected to a neighbor subspace $\mathcal{P}_{(1,2)} \Gamma_0$ via three line segments: we first shift $w'$ towards $w_1$ while keeping the other parameters fixed and then move $a_1^1$ from $a_1$ to $0$ while keeping $a_1^1+ a_1^2=a_1$. The connectivity graph (bottom right) shows each subspace as an appropriately colored dot. (b) Case of a network with three hidden neurons with parameters $(w_1, w', w_2, a_1, 0, a_2)$ that is reducible to a network with two hidden neurons. $\Gamma_0$ is connected to any other subspace $\mathcal{P}_\pi \Gamma_0$ through transitions from one neighbor to the next. Note that there are $T(2, 3) = 12$ subspaces. (c) The connectivity graph of subspaces for the expansion $3 \to 4$, there are $T(3, 4) = 60$ subspaces ($24$ blue and $36$ green), where each blue subspace is connected to three green subspaces and each green subspace is connected to two blue subspaces.
  • Figure 5: Left: The function $\sigma_{\alpha,\gamma}(x) = \sigma_{\text{soft}}(x) + \alpha \sigma_{\text{sig}}(\gamma x)$ satisfies the technical condition of Theorem \ref{['thm:all-global-minima']}. With this activation function, data is generated by a teacher network of width 4. All 50 student networks with width 10 find a global minimum by reaching loss values below $10^{-16}$. Right: The 500 = 50$\times$10 hidden neurons of all the 50 student networks are classified as copies of teacher neurons or zero-type neurons with vanishing sum of output weights. The zero-type neurons are further classified according to group size: there are 34 neurons with vanishing output weight (group size 1), 54 neurons that have a partner neuron with the same input weights and the sum of output weights equal to 0 (group size 2) etc. All zero-type neurons and replications of weight vectors can be pruned.
  • ...and 7 more figures

Theorems & Definitions (52)

  • Definition 2.1
  • Definition 2.2
  • Lemma 2.1
  • Remark 2.1
  • Definition 3.1
  • Definition 3.2
  • Definition 3.3
  • Theorem 3.1
  • Corollary 4.1
  • Theorem 4.2
  • ...and 42 more