ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation

TL;DR

ShiftAddAug tackles the challenge of deploying multiplication-free neural networks on resource-constrained devices by embedding a tiny Shift/Add-based network inside a larger multiplicative network during training to gain supervision. It introduces heterogeneous weight sharing and a two-stage neural architecture search to design stronger, deployment-ready tiny models that maintain hardware efficiency. Across image classification and semantic segmentation, ShiftAddAug yields up to 4.95 percentage points accuracy gains on CIFAR-100 and improvements over state-of-the-art multiplication-free methods, while incurring no inference overhead. The approach also highlights training-time costs and memory increases as limitations, supported by comprehensive ablations demonstrating the contributions of hybrid augmentation and weight-remapping strategies.

Abstract

Operators devoid of multiplication, such as Shift and Add, have gained prominence for their compatibility with hardware. However, neural networks (NNs) employing these operators typically exhibit lower accuracy compared to conventional NNs with identical structures. ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving performance without any inference overhead. It puts a ShiftAdd tiny NN into a large multiplicative model and encourages it to be trained as a sub-model to obtain additional supervision. In order to solve the weight discrepancy problem between hybrid operators, a new weight sharing method is proposed. Additionally, a novel two stage neural architecture search is used to obtain better augmentation effects for smaller but stronger multiplication-free tiny neural networks. The superiority of ShiftAddAug is validated through experiments in image classification and semantic segmentation, consistently delivering noteworthy enhancements. Remarkably, it secures up to a 4.95% increase in accuracy on the CIFAR100 compared to its directly trained counterparts, even surpassing the performance of multiplicative NNs.

Figures (7)

• Figure 1: ShiftAddAug augments weak operators with stronger ones. In this framework, pink indicates the multiplication-free kernels designated for the target model (either Shift or Add), while orange represents the multiplicative kernels of the augmented portion. During training, a wide weight is maintained, with the initial $n$ channels processed in a multiplication-free manner and the subsequent channels utilizing multiplicative operations. The weights of both are updated but only the multiplication-free part is exported for deployment. Therefore, the important weights in the wide weight will be reordered into the multiplication-free part. Obtained tiny models have higher accuracy (up to 4.95%) than their directly trained counterparts.
• Figure 2: Weight distribution of different convolution operators in MobileNetV2-w0.35. Inconsistent weight distribution leads to discrepancy, making weight sharing difficult.
• Figure 3: Weight remapping strategy creates mappings between different weight distributions, making weight sharing workable. $cpf_g$ for cumulative probability function (CPF) for Gaussian distribution and $cpf_l$ CPF for Laplace distribution
• Figure 4: Light orange block for $\texttt{MConv}$; pink block for $\texttt{MFConv}$; deep orange block for Depth Augmentation. (a).Search process: two-stage search. Find a SubNet as depth augmented one and then further cut out a tiny TargetNet on it for development. Start with multiplication and gradually convert the TargetNet to multiplication-free during training. (b).Methods to augment. Augment Width: use $\texttt{MConv}$ to widen the $\texttt{MFConv}$ channel; Augment Expand: increase expand ratio of depthwise separable convolution; Augment Depth: Augmented blocks for depth. Only participate in training and will not be exported; Block Mutation: start with $\texttt{MConv}$, mutate the block into $\texttt{MFConv}$ during training.
• Figure 5: Accuracy and energy cost of ShiftAddAug over SOTA manually designed multiplication-free model and tiny multiplicative models. Tested on CIFAR-100/10.