ShiftAddNet: A Hardware-Inspired Deep Network

TL;DR

ShiftAddNet introduces a hardware-inspired, multiplication-free neural architecture that substitutes all multiplications with bit-shift and additive operations, enabling energy-efficient training and inference while preserving expressive capacity. By combining coarse-grained shift layers with fine-grained add layers, it achieves competitive accuracy with substantial energy savings, validated by FPGA measurements and extensive ablations. The authors also propose a fixed-shift variant that further improves efficiency and quantization robustness, and demonstrate strong results on CIFAR benchmarks and IoT domain tasks, including adaptation and pruning scenarios. This work highlights a practical path toward deployment of DNNs on resource-constrained edge devices, with real energy measurements and robust performance across diverse datasets.

Abstract

Multiplication (e.g., convolution) is arguably a cornerstone of modern deep neural networks (DNNs). However, intensive multiplications cause expensive resource costs that challenge DNNs' deployment on resource-constrained edge devices, driving several attempts for multiplication-less deep networks. This paper presented ShiftAddNet, whose main inspiration is drawn from a common practice in energy-efficient hardware implementation, that is, multiplication can be instead performed with additions and logical bit-shifts. We leverage this idea to explicitly parameterize deep networks in this way, yielding a new type of deep network that involves only bit-shift and additive weight layers. This hardware-inspired ShiftAddNet immediately leads to both energy-efficient inference and training, without compromising the expressive capacity compared to standard DNNs. The two complementary operation types (bit-shift and add) additionally enable finer-grained control of the model's learning capacity, leading to more flexible trade-off between accuracy and (training) efficiency, as well as improved robustness to quantization and pruning. We conduct extensive experiments and ablation studies, all backed up by our FPGA-based ShiftAddNet implementation and energy measurements. Compared to existing DNNs or other multiplication-less models, ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies. Codes and pre-trained models are available at https://github.com/RICE-EIC/ShiftAddNet.

Figures (8)

• Figure 1: Illustrating the overview structure of ShiftAddNet.
• Figure 2: Tesing accuracy vs. energy cost of ShiftAddNet over AdderNet chen2019addernet (add only), DeepShift elhoushi2019deepshift (shift only), and multiplication-based ConvNet frankle2018the, using ResNet-20 and VGG19-small models on CIFAR-10/100 and two IoT datasets.
• Figure 3: Testing accuracy's trajectories visualization for ShiftAddNet, AdderNet chen2019addernet, and DeepShift elhoushi2019deepshift versus both training epochs and energy costs when evaluated on ResNet-20 with CIFAR-10.
• Figure 4: Testing accuracy vs. energy cost of ShiftAddNet with fixed shift layers over AdderNet chen2019addernet (add only), DeepShift elhoushi2019deepshift (shift only), and multiplication-based ConvNet frankle2018the, using the ResNet-20 and VGG19-small models on the CIFAR-10/100 and two IoT datasets.
• Figure 5: Testing accuracy vs. training epochs for the AdderNet chen2019addernet and pruned ShiftAddNets on ResNet-20 with CIFAR-10.