Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Quantization-Aware Knowledge Distillation

Kaiqi Zhao, Ming Zhao

TL;DR

A novel Self-Supervised Quantization-Aware Knowledge Distillation framework (SQAKD), which unifies the forward and backward dynamics of various quantization functions, making it flexible for incorporating the various QAT works and significantly improves the performance of state-of-the-art QAT works.

Abstract

Quantization-aware training (QAT) and Knowledge Distillation (KD) are combined to achieve competitive performance in creating low-bit deep learning models. However, existing works applying KD to QAT require tedious hyper-parameter tuning to balance the weights of different loss terms, assume the availability of labeled training data, and require complex, computationally intensive training procedures for good performance. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation (SQAKD) framework. SQAKD first unifies the forward and backward dynamics of various quantization functions, making it flexible for incorporating various QAT works. Then it formulates QAT as a co-optimization problem that simultaneously minimizes the KL-Loss between the full-precision and low-bit models for KD and the discretization error for quantization, without supervision from labels. A comprehensive evaluation shows that SQAKD substantially outperforms the state-of-the-art QAT and KD works for a variety of model architectures. Our code is at: https://github.com/kaiqi123/SQAKD.git.

Self-Supervised Quantization-Aware Knowledge Distillation

TL;DR

A novel Self-Supervised Quantization-Aware Knowledge Distillation framework (SQAKD), which unifies the forward and backward dynamics of various quantization functions, making it flexible for incorporating the various QAT works and significantly improves the performance of state-of-the-art QAT works.

Abstract

Quantization-aware training (QAT) and Knowledge Distillation (KD) are combined to achieve competitive performance in creating low-bit deep learning models. However, existing works applying KD to QAT require tedious hyper-parameter tuning to balance the weights of different loss terms, assume the availability of labeled training data, and require complex, computationally intensive training procedures for good performance. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation (SQAKD) framework. SQAKD first unifies the forward and backward dynamics of various quantization functions, making it flexible for incorporating various QAT works. Then it formulates QAT as a co-optimization problem that simultaneously minimizes the KL-Loss between the full-precision and low-bit models for KD and the discretization error for quantization, without supervision from labels. A comprehensive evaluation shows that SQAKD substantially outperforms the state-of-the-art QAT and KD works for a variety of model architectures. Our code is at: https://github.com/kaiqi123/SQAKD.git.
Paper Structure (25 sections, 6 equations, 10 figures, 12 tables, 1 algorithm)

This paper contains 25 sections, 6 equations, 10 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Methodology
  4. QAT as Constrained Optimization
  5. Analysis of KD in QAT
  6. Optimization via Self-Supervised KD
  7. Evaluation
  8. Improvements on SOTA QAT Methods
  9. Comparison with SOTA KD Methods
  10. Comparison with SOTA Methods Applying both QAT and KD
  11. Inference Speedup
  12. Ablation Study
  13. Conclusion
  14. Acknowledgments
  15. Experiment Setup
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Workflow of SQAKD.
  • Figure 2: Illustration of the evolution of (a) CE-Loss and (b) KL-Loss in each iteration during the training of 1-bit VGG-13 on CIFAR-100.
  • Figure 3: Top-1 test accuracy evolution of full-precision (FP) and quantized models using the standalone EWES and SQAKD integrating EWGS, during training.
  • Figure 4: (a) Top-1 test accuracy, (b) CE-Loss, and (c) KL-Loss of EWGS, FSP, and SQAKD in each epoch/iteration during training on 2-bit ResNet-20 with CIFAR-10.
  • Figure 5: 3D loss surface (a, b, c) and 2D contours (d, e, f) for full-precision and 2-bit ResNet-20 using SQAKD and EWGS on CIFAR-10.
  • ...and 5 more figures