Do deep neural networks utilize the weight space efficiently?
Onur Can Koyun, Behçet Uğur Töreyin
TL;DR
The paper tackles the high parameter burden of modern architectures by introducing a weight-space factorization that exploits the column-space and row-space of weight matrices. It recasts Transformer encoder and CNN bottleneck computations to reduce parameters, yielding roughly a 2× reduction with only minor accuracy degradation, demonstrated on ImageNet-1k with ViT and ResNet-50. Key formulations include $FFN(\mathbf{x}) = \mathbf{W}^T \mathcal{F}(\mathbf{W}\mathbf{x} + \boldsymbol{b_1}) + \boldsymbol{b_2}$ and $\text{Bottleneck}(\mathbf{x}) = \mathbf{W}^T \mathcal{G}(\mathbf{W}\mathbf{x})$, as well as $\\text{Bottleneck}(\mathbf{x_1}) = \mathbf{W}^T\\mathcal{G}_1(\mathbf{W}\mathbf{x_1}) + \mathbf{x_1}$ (with staged weight sharing). Experiments show ViT-PE and ResNet50-PE achieving competitive top-1 accuracy with about half the parameters, indicating strong practical potential for edge and mobile deployments. Overall, the approach offers a simple, generalizable path to parameter-efficient deep learning across attention-based and convolutional components.
Abstract
Deep learning models like Transformers and Convolutional Neural Networks (CNNs) have revolutionized various domains, but their parameter-intensive nature hampers deployment in resource-constrained settings. In this paper, we introduce a novel concept utilizes column space and row space of weight matrices, which allows for a substantial reduction in model parameters without compromising performance. Leveraging this paradigm, we achieve parameter-efficient deep learning models.. Our approach applies to both Bottleneck and Attention layers, effectively halving the parameters while incurring only minor performance degradation. Extensive experiments conducted on the ImageNet dataset with ViT and ResNet50 demonstrate the effectiveness of our method, showcasing competitive performance when compared to traditional models. This approach not only addresses the pressing demand for parameter efficient deep learning solutions but also holds great promise for practical deployment in real-world scenarios.