Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization

TL;DR

Low Rank Decomposition adds decomposed layers that can limit speedups, so the paper introduces Rank Optimization and Sequential Freezing to achieve substantial training and inference acceleration without aggressively reducing ranks. It leverages SVD for 1x1 and fully connected layers and Tucker for convolutions, and uses a data-driven rank search to identify an optimal rank $r$ around LRD estimates, plus a freezing strategy that reduces backpropagation cost. Experimental results on ResNet variants and Vision Transformer models across GPUs and NPUs show up to ~60% training speedup and ~37% inference speedup with only minor accuracy loss, with sequential freezing also accelerating convergence on CIFAR-10. These techniques are platform-agnostic and scalable across architectures, enabling practical deployment of LRD-compressed models.

Abstract

Low Rank Decomposition (LRD) is a model compression technique applied to the weight tensors of deep learning models in order to reduce the number of trainable parameters and computational complexity. However, due to high number of new layers added to the architecture after applying LRD, it may not lead to a high training/inference acceleration if the decomposition ranks are not small enough. The issue is that using small ranks increases the risk of significant accuracy drop after decomposition. In this paper, we propose two techniques for accelerating low rank decomposed models without requiring to use small ranks for decomposition. These methods include rank optimization and sequential freezing of decomposed layers. We perform experiments on both convolutional and transformer-based models. Experiments show that these techniques can improve the model throughput up to 60% during training and 37% during inference when combined together while preserving the accuracy close to that of the original models

Figures (3)

• Figure 1: Low Rank Decomposition of 1x1 and 3x3 convolutional layers. Note that FC layers are treated the same as 1x1 Conv layers.
• Figure 2: Effect of rank selection on throughput of a 3x3 Conv layer in ResNet-152 with dimensions [512, 512, 3, 3] when decomposed using Tucker2 method with different ranks
• Figure 3: Fine-tuning of ResNet-50 using sequential layer freezing (black line) vs regular freezing (red line) on CIFAR-10 dataset.