Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting

TL;DR

A sample-independent critical lifting operator is introduced that associates a parameter of one network with a set of parameters of another, thus defining sample-dependent and sample-independent lifted critical points.

Abstract

This paper investigates the sample dependence of critical points for neural networks. We introduce a sample-independent critical lifting operator that associates a parameter of one network with a set of parameters of another, thus defining sample-dependent and sample-independent lifted critical points. We then show by example that previously studied critical embeddings do not capture all sample-independent lifted critical points. Finally, we demonstrate the existence of sample-dependent lifted critical points for sufficiently large sample sizes and prove that saddles appear among them.

Figures (3)

• Figure 1: Plot of the vector field $(a_1, a_2) \mapsto \left( \frac{\partial{R}}{\partial{a_1}}(a_1, \bar{w}, a_2, \bar{w}), \frac{3}{a_1} \frac{\partial{R}}{\partial{w_1}}(a_1, \bar{w}, a_2, \bar{w}) \right)$ for $(a_1, a_2) \in (0.1, 0.9)^2$ with respect to $(\varepsilon_i(-4))_{i=1}^4$ (left), $(\varepsilon_i(0))_{i=1}^4$ (middle) and $(\varepsilon_i(3))_{i=1}^4$. In all three figures, the vector field vanishes approximately along the line $\{a_1 + a_2 = 1\}$, indicating that the parameters produced by splitting embeddings are sample-independent saddles.
• Figure 2: Contour plot of the loss function along the $(w_2, a_2)$-plane with respect to $(\varepsilon_i(0))_{i=1}^4$. The points, marked in red, are approximately $(0, 0)$ (left), $(0.1236, 0)$ (middle) and $(1.0258, 0)$ (right). They correspond to the critical points $(1, \bar{w}, 0, 0), (1, \bar{w}, 0, 0.1236), (1, \bar{w}, 0, 1.0258)$ in $E'$, respectively. From the level curves we can see that these three points are all saddles. Note that in the rightmost figure $w_2$-axis is scaled by 10 for illustration purpose.
• Figure 3: The zero set of $\varphi(t) = \sum_{i=1}^4 \varepsilon_i(t) \mathrm{tanh}(w x_i)$ for $(t,w) \in (-0.5, 0.5) \times (-0.8, 0.8)$. The blue curve minus the origin, which arises when $t$ ranges approximately from $-0.05$ to $0.3$, is locally the graph of a non-constant function in $t$. This indicates that there is a sample-dependent lifted critical point for each such $t$. Also note that the grey curve $\{(0,t)\}$ indicates a sample-independent lifted critical point $(1, \bar{w}, 0, 0)$. It arises due to the fact that $\tanh(0) = 0$.

Theorems & Definitions (43)

• Definition 3.1: wider/narrower neural network
• Remark 3.1
• Definition 4.1: sample-independent critical lifting
• Definition 4.2: sample-dependent/independent lifted critical points
• Remark 4.1
• Proposition 4.1.1: critical embeddings produce sample-independent lifted critical points
• Remark 4.2
• Proposition 4.2.1: saddles, one hidden layer
• Remark 4.3
• proof