Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee

Chenyang Li, Jihoon Chung, Mengnan Du, Haimin Wang, Xianlian Zhou, Bo Shen

TL;DR

This work introduces a holistic, nonconvex optimization framework for model compression that unifies low-rank tensorization and weight pruning via a neural-network block coordinate descent (NN-BCD) algorithm. NN-BCD is gradient-free, leverages proximal updates, and handles non-differentiable and nonconvex components, with global convergence to a critical point at a rate of $O(1/k)$ under mild conditions through the Kurdyka–Łojasiewicz property. The authors provide detailed algorithmic steps, subproblem solutions, and convergence proofs, enabling efficient, scalable training for TT-based layers and pruned networks. Empirically, NN-BCD achieves strong performance on tensorized CNNs and MLPs (MNIST, HAR) and on weight-pruned LeNet-300-100 with high sparsity, often matching or surpassing uncompressed baselines at substantial compression. The approach offers a practical pathway to deploy compact, high-performing neural networks in resource-constrained environments.

Abstract

Model compression is a crucial part of deploying neural networks (NNs), especially when the memory and storage of computing devices are limited in many applications. This paper focuses on two model compression techniques: low-rank approximation and weight pruning in neural networks, which are very popular nowadays. However, training NN with low-rank approximation and weight pruning always suffers significant accuracy loss and convergence issues. In this paper, a holistic framework is proposed for model compression from a novel perspective of nonconvex optimization by designing an appropriate objective function. Then, we introduce NN-BCD, a block coordinate descent (BCD) algorithm to solve the nonconvex optimization. One advantage of our algorithm is that an efficient iteration scheme can be derived with closed-form, which is gradient-free. Therefore, our algorithm will not suffer from vanishing/exploding gradient problems. Furthermore, with the Kurdyka-Łojasiewicz (KŁ) property of our objective function, we show that our algorithm globally converges to a critical point at the rate of O(1/k), where k denotes the number of iterations. Lastly, extensive experiments with tensor train decomposition and weight pruning demonstrate the efficiency and superior performance of the proposed framework. Our code implementation is available at https://github.com/ChenyangLi-97/NN-BCD

On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee

TL;DR

This work introduces a holistic, nonconvex optimization framework for model compression that unifies low-rank tensorization and weight pruning via a neural-network block coordinate descent (NN-BCD) algorithm. NN-BCD is gradient-free, leverages proximal updates, and handles non-differentiable and nonconvex components, with global convergence to a critical point at a rate of under mild conditions through the Kurdyka–Łojasiewicz property. The authors provide detailed algorithmic steps, subproblem solutions, and convergence proofs, enabling efficient, scalable training for TT-based layers and pruned networks. Empirically, NN-BCD achieves strong performance on tensorized CNNs and MLPs (MNIST, HAR) and on weight-pruned LeNet-300-100 with high sparsity, often matching or surpassing uncompressed baselines at substantial compression. The approach offers a practical pathway to deploy compact, high-performing neural networks in resource-constrained environments.

Abstract

Model compression is a crucial part of deploying neural networks (NNs), especially when the memory and storage of computing devices are limited in many applications. This paper focuses on two model compression techniques: low-rank approximation and weight pruning in neural networks, which are very popular nowadays. However, training NN with low-rank approximation and weight pruning always suffers significant accuracy loss and convergence issues. In this paper, a holistic framework is proposed for model compression from a novel perspective of nonconvex optimization by designing an appropriate objective function. Then, we introduce NN-BCD, a block coordinate descent (BCD) algorithm to solve the nonconvex optimization. One advantage of our algorithm is that an efficient iteration scheme can be derived with closed-form, which is gradient-free. Therefore, our algorithm will not suffer from vanishing/exploding gradient problems. Furthermore, with the Kurdyka-Łojasiewicz (KŁ) property of our objective function, we show that our algorithm globally converges to a critical point at the rate of O(1/k), where k denotes the number of iterations. Lastly, extensive experiments with tensor train decomposition and weight pruning demonstrate the efficiency and superior performance of the proposed framework. Our code implementation is available at https://github.com/ChenyangLi-97/NN-BCD
Paper Structure (25 sections, 7 theorems, 76 equations, 15 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 7 theorems, 76 equations, 15 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Notation and Research Background
  3. Notations
  4. Tensor Train Fully-Connected Layer
  5. Weight Pruning
  6. Methodology
  7. Problem Formulation
  8. Neural Network BCD Algorithm
  9. Convergence Analysis
  10. Experiments
  11. Experiments on Tensorized CNN
  12. Experiments on Tensorized MLP
  13. Experiments on Weight Pruning
  14. Conclusion
  15. Kernel K Update for Convolutional Layers
  16. ...and 10 more sections

Key Result

Lemma 5

Suppose $\alpha,\gamma,\rho,\tau>0$ and $\left\{\mathcal{P}^{k}\right\}_{k \in \mathbb{N}}$ is the sequence generated by the NN-BCD algorithm alg: NN-BCD. Then we have where $\lambda=\min \left\{ \alpha, \gamma+\rho,\tau\right\}/2$.

Figures (15)

  • Figure 1: The proposed framework of NN training for model compression.
  • Figure 2: The boxplots among ten repetitions with different compression ratios (CNN MNIST): (a) training loss; (b) training accuracy; (c) test accuracy.
  • Figure 3: The convergence analysis of NN-BCD algorithm with different compression ratios (CNN MNIST): (a) training loss; (b) training error rate; (c) test error rate. The Y-axis is in the log scale.
  • Figure 4: Effect of different hyperparameters and weight initialization of NN-BCD algorithm (CNN MNIST) when CR = 0.1784: (a) Effect of hyperparameters of NN-BCD; (b) Stability of different weight initialization methods.
  • Figure 5: The boxplots among ten repetitions with different compression ratios (MLP-4 HAR): (a) training loss; (b) training accuracy; (c) test accuracy.
  • ...and 10 more figures

Theorems & Definitions (16)

  • Remark 1
  • Remark 2: Advantages of Formulation \ref{['eq: final formulation']}
  • Definition 3: Critical point attouch2009convergenceattouch2010proximal
  • Definition 4: Global convergence petrovai2017globalxu2018convergence
  • Lemma 5: Sufficient Decrease Property
  • Lemma 6: Subgradient Bound
  • Theorem 7: Global Convergence of NN-BCD
  • Lemma 8
  • Lemma 9
  • Definition 10: Subdifferentials attouch2009convergenceattouch2010proximal
  • ...and 6 more