Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AdaQAT: Adaptive Bit-Width Quantization-Aware Training

Cédric Gernigon, Silviu-Ioan Filip, Olivier Sentieys, Clément Coggiola, Mickael Bruno

TL;DR

AdaQAT tackles the challenge of efficient inference for large DNNs on edge devices by learning mixed-precision bit-widths during training. It introduces real-valued bit-widths $N_w$ and $N_a$ that are continuously updated via gradient-based optimization, with actual bit-widths given by $\lceil N_w\rceil$ and $\lceil N_a\rceil$, and a total loss that jointly optimizes task accuracy and hardware cost. The method builds on DoReFa weight quantization and PACT activations, frames bit-width optimization in a differentiable-like objective using finite-difference gradients, and uses a practical strategy to handle oscillations in bit-width values. Empirical results on CIFAR-10 and ImageNet demonstrate competitive accuracy while achieving substantial reductions in BitOPs, validating AdaQAT’s potential for flexible, end-to-end mixed-precision quantization.

Abstract

Large-scale deep neural networks (DNNs) have achieved remarkable success in many application scenarios. However, high computational complexity and energy costs of modern DNNs make their deployment on edge devices challenging. Model quantization is a common approach to deal with deployment constraints, but searching for optimized bit-widths can be challenging. In this work, we present Adaptive Bit-Width Quantization Aware Training (AdaQAT), a learning-based method that automatically optimizes weight and activation signal bit-widths during training for more efficient DNN inference. We use relaxed real-valued bit-widths that are updated using a gradient descent rule, but are otherwise discretized for all quantization operations. The result is a simple and flexible QAT approach for mixed-precision uniform quantization problems. Compared to other methods that are generally designed to be run on a pretrained network, AdaQAT works well in both training from scratch and fine-tuning scenarios.Initial results on the CIFAR-10 and ImageNet datasets using ResNet20 and ResNet18 models, respectively, indicate that our method is competitive with other state-of-the-art mixed-precision quantization approaches.

AdaQAT: Adaptive Bit-Width Quantization-Aware Training

TL;DR

AdaQAT tackles the challenge of efficient inference for large DNNs on edge devices by learning mixed-precision bit-widths during training. It introduces real-valued bit-widths and that are continuously updated via gradient-based optimization, with actual bit-widths given by and , and a total loss that jointly optimizes task accuracy and hardware cost. The method builds on DoReFa weight quantization and PACT activations, frames bit-width optimization in a differentiable-like objective using finite-difference gradients, and uses a practical strategy to handle oscillations in bit-width values. Empirical results on CIFAR-10 and ImageNet demonstrate competitive accuracy while achieving substantial reductions in BitOPs, validating AdaQAT’s potential for flexible, end-to-end mixed-precision quantization.

Abstract

Large-scale deep neural networks (DNNs) have achieved remarkable success in many application scenarios. However, high computational complexity and energy costs of modern DNNs make their deployment on edge devices challenging. Model quantization is a common approach to deal with deployment constraints, but searching for optimized bit-widths can be challenging. In this work, we present Adaptive Bit-Width Quantization Aware Training (AdaQAT), a learning-based method that automatically optimizes weight and activation signal bit-widths during training for more efficient DNN inference. We use relaxed real-valued bit-widths that are updated using a gradient descent rule, but are otherwise discretized for all quantization operations. The result is a simple and flexible QAT approach for mixed-precision uniform quantization problems. Compared to other methods that are generally designed to be run on a pretrained network, AdaQAT works well in both training from scratch and fine-tuning scenarios.Initial results on the CIFAR-10 and ImageNet datasets using ResNet20 and ResNet18 models, respectively, indicate that our method is competitive with other state-of-the-art mixed-precision quantization approaches.
Paper Structure (16 sections, 10 equations, 1 figure, 3 tables)

This paper contains 16 sections, 10 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Quantization-Aware Training
  4. Bit-Width Search Strategies
  5. Method
  6. Quantization background
  7. Objective Function
  8. Bit-Width Gradients & Parameter Updates
  9. Experiments
  10. Experimental Setup
  11. Datasets
  12. Networks
  13. Training Settings
  14. Comparison with State-of-the-Art Methods
  15. Balancing Parameter Impact
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Example of applying our approach with a ResNet20 network on the CIFAR-10 dataset. It showcases the evolution of the train accuracy with respect to updating the bit-width parameters $\left\lceil N_\mathbf{w}\right\rceil$ and $\left\lceil N_\mathbf{a}\right\rceil$ and how an oscillatory pattern can form (here, for the weight bit-width $\left\lceil N_\mathbf{w}\right\rceil$). When oscillations appear, we fix the value of the corresponding bit-width to the largest of the two oscillation points for the rest of the QAT process, considering that it has converged.