Alternating Multi-bit Quantization for Recurrent Neural Networks

TL;DR

Problem: Recurrent neural networks are large and slow, hindering deployment on resource-limited devices and high-throughput servers. Approach: The authors propose Alternating Multi-bit Quantization (AMQ) to quantize weights and activations to multi-bit binary codes using an alternating optimization between coefficients and binary codes, with efficient binary-search-tree-based code assignment. Findings: 2-bit quantization yields substantial memory and CPU speedups with moderate accuracy loss; 3-bit quantization achieves near-lossless accuracy with further memory and speedups, outperforming prior quantization methods; the approach generalizes to image classification and feedforward nets. Impact: Provides a practical, scalable method for deploying compact, fast RNNs on diverse hardware with broad applicability.

Abstract

Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied. We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration. Both results beat the exiting quantization works with large margins. We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves excellent performance.

Figures (2)

• Figure 1: Illustration of the optimal $2$-bit quantization when $\alpha_1$ and $\alpha_2$ ($\alpha_1 \geq \alpha_2$) are known in advance. The values are quantized into $-\alpha_1 - \alpha_2$, $-\alpha_1 + \alpha_2$, $\alpha_1 - \alpha_2$, and $\alpha_1 + \alpha_2$, respectively. And the partition intervals are optimally separated by the middle points of adjacent quantization codes, i.e., $-\alpha_1$, $0$, and $\alpha_1$, correspondingly.
• Figure 2: Illustration of binary search tree to determine the optimal quantization.