Deep Neural Network Initialization with Sparsity Inducing Activations

TL;DR

The large width Gaussian process limit is used to analyze the behaviour of nonlinear activations that induce sparsity in the hidden outputs and shows that the proposed magnitude clipped sparsifying activations can be trained with training and test fractional sparsity as high as 85\% while retaining close to full accuracy.

Abstract

Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly important as network sizes continue to grow and their application becomes more widespread. Here we use the large width Gaussian process limit to analyze the behaviour, at random initialization, of nonlinear activations that induce sparsity in the hidden outputs. A previously unreported form of training instability is proven for arguably two of the most natural candidates for hidden layer sparsification; those being a shifted ReLU ($φ(x)=\max(0, x-τ)$ for $τ\ge 0$) and soft thresholding ($φ(x)=0$ for $|x|\leτ$ and $x-\text{sign}(x)τ$ for $|x|>τ$). We show that this instability is overcome by clipping the nonlinear activation magnitude, at a level prescribed by the shape of the associated Gaussian process variance map. Numerical experiments verify the theory and show that the proposed magnitude clipped sparsifying activations can be trained with training and test fractional sparsity as high as 85\% while retaining close to full accuracy.

Figures (14)

• Figure 1: From left to right: ReLU (i.e. $\mathsf{ReLU}_{\tau}$ with $\tau=0$), $\mathsf{ReLU}_{\tau}$ ($\tau=1$), $\mathsf{ST}_{\tau}$ ($\tau=1$), $\mathsf{CReLU}_{\tau,m}$ ($\tau=1, m=1$), $\mathsf{CST}_{\tau,m}$ ($\tau=1, m=1$)
• Figure 2: Variance maps for $\mathsf{ReLU}_{\tau}$ and $\mathsf{ST}_{\tau}$ with $(\sigma_w,\sigma_b)$ on the EoC, for different values of $\tau$. Here $q^*=1$ is used to compute $\chi_{1,\mathsf{ReLU}_{\tau}}$. The dashed line is the identity map. The curves for $\mathsf{ReLU}_{\tau}$ and $\mathsf{ST}_{\tau}$ overlap exactly for a given $\tau$. Note, however, that a fixed value of $\tau$ corresponds to substantially different activation sparsities for $\mathsf{ReLU}_{\tau}$ and $\mathsf{ST}_{\tau}$. Figure \ref{['fig:variance map sparsity']} in App. \ref{['sec: non max variants experiments']} compares $V(q)$ for fixed output sparsities.
• Figure 3: Plots of $V'_{\phi}(m)$ (upper) and $V"_{\phi}(m)$ (lower) and at $q^*=1$ for $\mathsf{CReLU}_{\tau,m}$ (left) and $\mathsf{CST}_{\tau,m}$ (right) with for different fractional sparsities. For $V'_{\phi}(m)$ the horizontal dashed black lines are plotted at 0.5, 0.6, 0.7, 0.8, 0.9.
• Figure 4: Variance maps for $\mathsf{CReLU}_{\tau,m}$ in (a) and $\mathsf{CST}_{\tau,m}$ in (b) at 85% sparsity and $m$ chosen such that even though $V'_\phi(q^*)<1$ the network is unstable to train. For $\mathsf{CReLU}_{\tau,m}$ training fails due to $V_{\phi}(q)$ being approximately $q$ for a sufficiently large region around $q^*$ that the effective value of $\chi_1$ varies above 1 which results in exploding gradients. For $\mathsf{CST}_{\tau,m}$ training fails for the same reason of effective $\chi_1>1$, which occurs even more dramatically as $V_{\phi}(q)$ exceeds $q$ and has a second, unstable, fixed point as well as another stable fixed point, but with $\chi_1$ substantially greater than 1.
• Figure 5: Gradient norms per layer for the first 15 steps of a failed training run of DNNs on MNIST.