Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

TL;DR

This paper presents a theoretically-justified technique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation.

Abstract

Deep Neural Networks (DNNs) have achieved remarkable success in addressing many previously unsolvable tasks. However, the storage and computational requirements associated with DNNs pose a challenge for deploying these trained models on resource-limited devices. Therefore, a plethora of compression and pruning techniques have been proposed in recent years. Low-rank decomposition techniques are among the approaches most utilized to address this problem. Compared to post-training compression, compression-promoted training is still under-explored. In this paper, we present a theoretically-justified technique termed Low-Rank Induced Training (LoRITa), that promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. This is achieved without the need to change the structure at inference time or require constrained and/or additional optimization, other than the standard weight decay regularization. Moreover, LoRITa eliminates the need to (i) initialize with pre-trained models, (ii) specify rank selection prior to training, and (iii) compute SVD in each iteration. Our experimental results (i) demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/100 and ImageNet on Convolutional Neural Networks, and (ii) illustrate that we achieve either competitive or state-of-the-art results when compared to leading structured pruning and low-rank training methods in terms of FLOPs and parameters drop. Our code is available at \url{https://github.com/XitongSystem/LoRITa/tree/main}.

Figures (5)

• Figure 2: Illustrative block diagram showcasing the compression (top), training (middle), and inference (bottom) for every convolution layer in CNNs. Here, '$*$' denotes the convolution operator.
• Figure 3: Results of the FCNs in Table \ref{['tab:overall_test_accuracy no aug']}. The top (resp. bottom) row corresponds to applying the Local (resp. Global) SVT. $N=1$ results represent the baseline, whereas $N>1$ results are for the LoRITa-trained models.
• Figure 4: Low-rank compression results of the considered CNNs using the CIFAR10 (left) and the CIFAR100 (right) datasets for the settings in Table \ref{['tab:overall_test_accuracy with aug']}. $N=1$ denotes the baseline, whereas $N=2$ represents our method.
• Figure 5: Low-rank compression of ViT models with data augmentation with varied attention heads and layers. The (top) and (bottom) plots correspond to the settings of Table \ref{['tab:overall_test_accuracy with aug']} and Table \ref{['tab:overall_test_accuracy no aug']}, respectively. $N=1$ denotes the baseline, while $N=2$ and $N=3$ represent our method.
• Figure 6: Empirically showing the faster decay of singular values of the first two weight matrices (layer 0 (left) and layer 1 (right)) of the standard model ($N=1$) vs. LoRITa-trained models ($N=2$ and $N=3$) using the FCN8 architecture of Table \ref{['tab:overall_test_accuracy no aug']}.

Theorems & Definitions (8)

• Remark 4.1
• Proposition 4.2
• Remark 4.3
• Remark 4.4
• Proposition 4.5
• Remark 4.6
• proof : Proof of Proposition \ref{['prop: main']}
• proof : Proof of Proposition \ref{['prop2']}