Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit

Chang Gao, Jianfei Chen, Kang Zhao, Jiaqi Wang, Liping Jing

TL;DR

This work pushes fully quantized training toward the 1-bit limit by developing a theory of FQT under Adam and SGD, showing gradient variance critically governs convergence and that Adam better tolerates low-bitwidth. It introduces Activation Gradient Pruning (AGP) to reduce gradient-variance by selectively retaining high-range gradient groups, and Sample Channel joint Quantization (SCQ) to tailor quantization for weight and activation gradients to hardware constraints. The 1-bit FQT algorithm combines sign-based forward propagation with unbiased, per-group gradient quantization and a hardware-friendly backpropagation path, achieving practical acceleration via AGP and SCQ. Empirically, the method delivers meaningful speedups (up to 5.13×) and competitive transfer-learning accuracy across CV and NLP tasks, while noting limitations in training-from-scratch scenarios and broader applicability to architectures beyond CNNs.

Abstract

Fully quantized training (FQT) accelerates the training of deep neural networks by quantizing the activations, weights, and gradients into lower precision. To explore the ultimate limit of FQT (the lowest achievable precision), we make a first attempt to 1-bit FQT. We provide a theoretical analysis of FQT based on Adam and SGD, revealing that the gradient variance influences the convergence of FQT. Building on these theoretical results, we introduce an Activation Gradient Pruning (AGP) strategy. The strategy leverages the heterogeneity of gradients by pruning less informative gradients and enhancing the numerical precision of remaining gradients to mitigate gradient variance. Additionally, we propose Sample Channel joint Quantization (SCQ), which utilizes different quantization strategies in the computation of weight gradients and activation gradients to ensure that the method is friendly to low-bitwidth hardware. Finally, we present a framework to deploy our algorithm. For fine-tuning VGGNet-16 and ResNet-18 on multiple datasets, our algorithm achieves an average accuracy improvement of approximately 6%, compared to per-sample quantization. Moreover, our training speedup can reach a maximum of 5.13x compared to full precision training.

1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit

TL;DR

This work pushes fully quantized training toward the 1-bit limit by developing a theory of FQT under Adam and SGD, showing gradient variance critically governs convergence and that Adam better tolerates low-bitwidth. It introduces Activation Gradient Pruning (AGP) to reduce gradient-variance by selectively retaining high-range gradient groups, and Sample Channel joint Quantization (SCQ) to tailor quantization for weight and activation gradients to hardware constraints. The 1-bit FQT algorithm combines sign-based forward propagation with unbiased, per-group gradient quantization and a hardware-friendly backpropagation path, achieving practical acceleration via AGP and SCQ. Empirically, the method delivers meaningful speedups (up to 5.13×) and competitive transfer-learning accuracy across CV and NLP tasks, while noting limitations in training-from-scratch scenarios and broader applicability to architectures beyond CNNs.

Abstract

Fully quantized training (FQT) accelerates the training of deep neural networks by quantizing the activations, weights, and gradients into lower precision. To explore the ultimate limit of FQT (the lowest achievable precision), we make a first attempt to 1-bit FQT. We provide a theoretical analysis of FQT based on Adam and SGD, revealing that the gradient variance influences the convergence of FQT. Building on these theoretical results, we introduce an Activation Gradient Pruning (AGP) strategy. The strategy leverages the heterogeneity of gradients by pruning less informative gradients and enhancing the numerical precision of remaining gradients to mitigate gradient variance. Additionally, we propose Sample Channel joint Quantization (SCQ), which utilizes different quantization strategies in the computation of weight gradients and activation gradients to ensure that the method is friendly to low-bitwidth hardware. Finally, we present a framework to deploy our algorithm. For fine-tuning VGGNet-16 and ResNet-18 on multiple datasets, our algorithm achieves an average accuracy improvement of approximately 6%, compared to per-sample quantization. Moreover, our training speedup can reach a maximum of 5.13x compared to full precision training.
Paper Structure (28 sections, 7 theorems, 37 equations, 10 figures, 5 tables, 2 algorithms)

This paper contains 28 sections, 7 theorems, 37 equations, 10 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Framework
  4. Quantized Training
  5. FQT with Unbiased Quantizer
  6. Theoretical Results
  7. Optimizer Impact on Convergence
  8. Quantizer Impact on Gradient Variance
  9. 1-bit FQT Algorithm
  10. Forward Propagation
  11. Backward Propagation
  12. Practical Acceleration
  13. Experiments
  14. Main Results
  15. Computational Efficiency
  16. ...and 13 more sections

Key Result

Theorem 4.3

If Assumption ass1 and ass2 holds, let $\alpha_t=\frac{\alpha}{\sqrt{t}}$ and the number of elements in the gradient is $d$. SGD achieves the following guarantee, for all $T \geq 1$. $R^{S G D}(T) \leq \frac{D^2}{2\alpha} + \frac{\alpha Td({\sigma}^2 + e^2)}{2}.$

Figures (10)

  • Figure 1: Gradient numerical precision ("bits") vs. test accuracy of VGGNet16 on CIFAR-10, trained with Adam and SGD.
  • Figure 2: Heterogeneity in a ResNet18’s gradients. (a) Heatmap of the per-group range at the conv2.1.2 layer; (b) Histogram of the gradient in a certain group.
  • Figure 3: The process of AGP and binary matrix multiplication. $BIT$ represents the bitwidth of full precision data. Here, we removed half of the groups, thus the bitwidth of the remaining groups is 2.
  • Figure 4: The computational graph of 1-bit FQT. The figure only shows the forward propagation of the last two layers and the backward propagation of the final layer.
  • Figure 5: Our method (left) vs. PSQ (right): Testing accuracy on VGGNet16 for CIFAR-10.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Theorem 4.3
  • Theorem 4.5
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • Lemma A.4
  • Proposition B.1