Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

TL;DR

The problem of storing an extensive number of random patterns is studied and it is found that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges.

Abstract

Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: while the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.

Figures (3)

• Figure 1: (Left panel) Typical solutions are isolated in 2 layer neural networks with binary weights. We plot the Franz-Parisi entropy $\mathcal{F}_{\text{FP}}$ as a function of the distance from a typical reference solution $\tilde{W}$ for the committee machine with ReLU activations (red line) and sign activations (blue line) in the limit of large number of neurons in the hidden layer $K$. We have used $\alpha=0.6$. For both curves there is a value of the distance for which the entropy becomes negative. This signals that typical solutions are isolated. (Right panel) Franz-Parisi entropy as a function of distance, normalized with respect to the unconstrained $\alpha=0$ case, for spherical weights and for $\alpha=1$.
• Figure 2: Numerical evidence of the greater robustness of the minima of ReLU transfer function (red line) compared with the sign one (blue line) using the large deviation analysis in the binary (left panel) setting and spherical setting (right panel). The exchange in the curves in both settings for $\alpha$ sufficiently large is due to the fact that the algorithmic threshold of the ReLU is reached before the corresponding one of the sign case.
• Figure 3: (Left panel) Dashed lines: theoretical stability curves for the typical solutions, for binary weights at $\alpha=0.4$. Solid lines: comparison between the numerical and theoretical stability distributions in the large deviation scenario, same $\alpha$. We have used $q_{1}=0.85$, $y=20$. (Right panel) Robustness of the reference configuration found by replicated simulated annealing when one pattern is perturbed by flipping a certain fraction $\eta$ of entries.