HAQ: Hardware-Aware Automated Quantization with Mixed Precision

TL;DR

HAQ introduces a hardware-aware automated quantization framework that uses reinforcement learning with direct hardware feedback to assign per-layer bitwidths for mixed-precision DNNs. By integrating a hardware accelerator as the environment, HAQ bypasses proxy metrics and discovers specialized policies for different platforms (edge/cloud, BISMO/BitFusion) under latency, energy, and model-size constraints. Empirical results on ImageNet with MobileNets show substantial latency and energy reductions with negligible accuracy loss relative to fixed 8-bit quantization, and the learned policies reveal meaningful hardware-architecture insights. The work demonstrates the value of automated, hardware-in-the-loop quantization for efficient deployment and provides design guidance for future hardware-software co-design in DNN inference.

Abstract

Model quantization is a widely used technique to compress and accelerate deep neural network (DNN) inference. Emergent DNN hardware accelerators begin to support mixed precision (1-8 bits) to further improve the computation efficiency, which raises a great challenge to find the optimal bitwidth for each layer: it requires domain experts to explore the vast design space trading off among accuracy, latency, energy, and model size, which is both time-consuming and sub-optimal. Conventional quantization algorithm ignores the different hardware architectures and quantizes all the layers in a uniform way. In this paper, we introduce the Hardware-Aware Automated Quantization (HAQ) framework which leverages the reinforcement learning to automatically determine the quantization policy, and we take the hardware accelerator's feedback in the design loop. Rather than relying on proxy signals such as FLOPs and model size, we employ a hardware simulator to generate direct feedback signals (latency and energy) to the RL agent. Compared with conventional methods, our framework is fully automated and can specialize the quantization policy for different neural network architectures and hardware architectures. Our framework effectively reduced the latency by 1.4-1.95x and the energy consumption by 1.9x with negligible loss of accuracy compared with the fixed bitwidth (8 bits) quantization. Our framework reveals that the optimal policies on different hardware architectures (i.e., edge and cloud architectures) under different resource constraints (i.e., latency, energy and model size) are drastically different. We interpreted the implication of different quantization policies, which offer insights for both neural network architecture design and hardware architecture design.

Figures (5)

• Figure 1: We need mixed precision for different layers. We quantize MobileNets Howard:2017wz to different number of bits (both weights and activations), and it lies on a better pareto curve (yellow) than fixed bit quantization (blue). The reason is that different layers have different redundancy and have different arithmetic intensity (OPs/byte) on the hardware, which advocates for using mixed precision for different layers.
• Figure 2: An overview of our Hardware-Aware Automated Quantization (HAQ) framework. We leverage the reinforcement learning to automatically search over the huge quantization design space with hardware in the loop. The agent propose an optimal bitwidth allocation policy given the amount of computation resources (i.e., latency, power, and model size). Our RL agent integrates the hardware accelerator into the exploration loop so that it can obtain the direct feedback from the hardware, instead of relying on indirect proxy signals.
• Figure 3: Quantization policy under latency constraints for MobileNet-V1. On edge accelerator, our RL agent allocates less activation bits to the depthwise convolutions, which echos that the depthwise convolutions are memory bounded and the activations dominates the memory access. On cloud accelerator, our agent allocates more bits to the depthwise convolutions and allocates less bits to the pointwise convolutions, as cloud device has more memory bandwidth and high parallelism, the network appears to be computation bounded.
• Figure 4: Quantization policy under latency constraints for MobileNet-V2 on BISMO. Similar to Figure \ref{['fig:bismo_v1']}, depthwise layer is assigned with fewer bits on the edge accelerator, and pointwise layer is assigned with fewer bits on the cloud accelerator.
• Figure 5: Quantization policy under model size constraints for MobileNet-V2. Our RL agent allocates more bits to the depthwise convolutions, since depthwise convolutions have fewer number of parameters.