Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantization-Aware Regularizers for Deep Neural Networks Compression

Dario Malchiodi, Mattia Ferraretto, Marco Frasca

TL;DR

The paper addresses the accuracy loss from weight quantization in deep neural networks by injecting quantization awareness into training. It proposes per‑layer regularizers that create quantization‑friendly weight clusters and learn quantization representatives jointly with weights, enabling end‑to‑end optimization with $\hat{\boldsymbol{W}}=\arg\min_{\boldsymbol{W}} \mathcal{L}(\boldsymbol{W},\mathcal{D})+\lambda\mathcal{R}(\boldsymbol{W})$. The approach offers static periodic and dynamic learnable basins for quantization as well as a weight‑sharing strategy, and demonstrates substantial pre‑tuning gains and competitive post‑tuning performance on CIFAR‑10 with AlexNet and VGG16. The findings suggest that quantization‑aware regularization can improve compression‑friendly training and facilitate efficient deployment on resource‑constrained hardware.

Abstract

Deep Neural Networks reached state-of-the-art performance across numerous domains, but this progress has come at the cost of increasingly large and over-parameterized models, posing serious challenges for deployment on resource-constrained devices. As a result, model compression has become essential, and -- among compression techniques -- weight quantization is largely used and particularly effective, yet it typically introduces a non-negligible accuracy drop. However, it is usually applied to already trained models, without influencing how the parameter space is explored during the learning phase. In contrast, we introduce per-layer regularization terms that drive weights to naturally form clusters during training, integrating quantization awareness directly into the optimization process. This reduces the accuracy loss typically associated with quantization methods while preserving their compression potential. Furthermore, in our framework quantization representatives become network parameters, marking, to the best of our knowledge, the first approach to embed quantization parameters directly into the backpropagation procedure. Experiments on CIFAR-10 with AlexNet and VGG16 models confirm the effectiveness of the proposed strategy.

Quantization-Aware Regularizers for Deep Neural Networks Compression

TL;DR

The paper addresses the accuracy loss from weight quantization in deep neural networks by injecting quantization awareness into training. It proposes per‑layer regularizers that create quantization‑friendly weight clusters and learn quantization representatives jointly with weights, enabling end‑to‑end optimization with . The approach offers static periodic and dynamic learnable basins for quantization as well as a weight‑sharing strategy, and demonstrates substantial pre‑tuning gains and competitive post‑tuning performance on CIFAR‑10 with AlexNet and VGG16. The findings suggest that quantization‑aware regularization can improve compression‑friendly training and facilitate efficient deployment on resource‑constrained hardware.

Abstract

Deep Neural Networks reached state-of-the-art performance across numerous domains, but this progress has come at the cost of increasingly large and over-parameterized models, posing serious challenges for deployment on resource-constrained devices. As a result, model compression has become essential, and -- among compression techniques -- weight quantization is largely used and particularly effective, yet it typically introduces a non-negligible accuracy drop. However, it is usually applied to already trained models, without influencing how the parameter space is explored during the learning phase. In contrast, we introduce per-layer regularization terms that drive weights to naturally form clusters during training, integrating quantization awareness directly into the optimization process. This reduces the accuracy loss typically associated with quantization methods while preserving their compression potential. Furthermore, in our framework quantization representatives become network parameters, marking, to the best of our knowledge, the first approach to embed quantization parameters directly into the backpropagation procedure. Experiments on CIFAR-10 with AlexNet and VGG16 models confirm the effectiveness of the proposed strategy.
Paper Structure (13 sections, 11 equations, 3 figures)

This paper contains 13 sections, 11 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Notation
  4. Preliminary Definitions
  5. Quantization‑Aware Weight Regularization
  6. Periodic Quantization‑Aware Weight Regularization
  7. Dynamic and Learnable Quantization‑Aware Weight Regularization
  8. Quantization via Weight Sharing
  9. Experiments
  10. Datasets
  11. Deep Models
  12. Performance Assessment
  13. Results

Figures (3)

  • Figure 1: $\rho_S$ (left) and $\rho_C$ (right) penalty functions in the interval $[w=-1, W=1]$ for $K=8$.
  • Figure 2: $\rho_M$ and $\rho_E$ functions in the interval [-1, 1] when minima ($\boldsymbol{u}$ assignment) are those denoted by the vertical dashed lines (K=8).
  • Figure 3: Accuracy ratio for AlexNet (first column) and for VGG16 (second column) on CIFAR-10 data. Rows correspond, from top to bottom to regularizing only convolutional layers, only dense layers, and all layers. Dotted lines denote a tie: all values above mean a gain for the quantization-aware regularization. The second axis shows the baseline accuracy.