Low-Rank Matrix Approximation for Neural Network Compression

TL;DR

This work tackles the memory and compute demands of deep neural networks by introducing Adaptive-Rank SVD (ARSVD), a per-layer rank selection strategy guided by spectral entropy. By defining $p_i=s_i/\sum_j s_j$, $H_{ ext{total}}=-\sum_i p_i\log p_i$, and choosing the smallest $k$ with $H(k)\ge \tau H_{ ext{total}}$, ARSVD adaptively compresses each weight matrix while preserving information-rich components. Empirical results on ResNet18, VGG16, and InceptionV3 across MNIST, CIFAR-10, and CIFAR-100 show substantial parameter and inference-time reductions with accuracy and F1 scores that are maintained or slightly improved compared to fixed-rank SVD baselines. The method demonstrates scalable, information-preserving compression and points to extensions for transformer models and learnable entropy thresholds, offering practical impact for deployment in resource-constrained settings.

Abstract

Deep Neural Networks (DNNs) have encountered an emerging deployment challenge due to large and expensive memory and computation requirements. In this paper, we present a new Adaptive-Rank Singular Value Decomposition (ARSVD) method that approximates the optimal rank for compressing weight matrices in neural networks using spectral entropy. Unlike conventional SVD-based methods that apply a fixed-rank truncation across all layers, ARSVD uses an adaptive selection of the rank per layer through the entropy distribution of its singular values. This approach ensures that each layer will retain a certain amount of its informational content, thereby reducing redundancy. Our method enables efficient, layer-wise compression, yielding improved performance with reduced space and time complexity compared to static-rank reduction techniques.

Figures (2)

• Figure 1: Comparison of the accuracy of the models across MNIST, CIFAR-10, and CIFAR-100; with standard SVD compression and with ARSVD compression. Accuracy metrics are normalized from [0,1]
• Figure 2: Comparison of the average inference times (seconds) of the models across MNIST, CIFAR-10, and CIFAR-100; with SVD compared to ARSVD