Learning both Weights and Connections for Efficient Neural Networks

TL;DR

The paper tackles the prohibitive storage and energy demands of large CNNs by introducing a three-step process that learns which connections are important, prunes the rest, and retrains the remaining sparse network. Through regularization choices, dropout adjustments, local co-adaptation, and iterative pruning, the method achieves substantial parameter reductions with negligible accuracy loss. Experiments on LeNet, AlexNet, and VGG-16 demonstrate 9x–13x reductions in parameters and sizable compute savings on ImageNet, with intact or even slightly improved accuracy. These results support deploying deep CNNs on embedded devices and motivate hardware that exploits sparsity for efficiency.

Abstract

Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems. Also, conventional networks fix the architecture before training starts; as a result, training cannot improve the architecture. To address these limitations, we describe a method to reduce the storage and computation required by neural networks by an order of magnitude without affecting their accuracy by learning only the important connections. Our method prunes redundant connections using a three-step method. First, we train the network to learn which connections are important. Next, we prune the unimportant connections. Finally, we retrain the network to fine tune the weights of the remaining connections. On the ImageNet dataset, our method reduced the number of parameters of AlexNet by a factor of 9x, from 61 million to 6.7 million, without incurring accuracy loss. Similar experiments with VGG-16 found that the number of parameters can be reduced by 13x, from 138 million to 10.3 million, again with no loss of accuracy.

Figures (7)

• Figure 1: Energy table for 45nm CMOS process markenergy. Memory access is 3 orders of magnitude more energy expensive than simple arithmetic.
• Figure 2: Three-Step Training Pipeline.
• Figure 3: Synapses and neurons before and after pruning.
• Figure 4: Visualization of the first FC layer's sparsity pattern of Lenet-300-100. It has a banded structure repeated 28 times, which correspond to the un-pruned parameters in the center of the images, since the digits are written in the center.
• Figure 5: Trade-off curve for parameter reduction and loss in top-5 accuracy. L1 regularization performs better than L2 at learning the connections without retraining, while L2 regularization performs better than L1 at retraining. Iterative pruning gives the best result.