And the Bit Goes Down: Revisiting the Quantization of Neural Networks

TL;DR

The paper tackles the memory bottleneck of high-performing ConvNets by introducing an activation-focused Product Quantization approach that minimizes the reconstruction error of layer outputs on in-domain data, using only unlabeled samples. It introduces a weighted k-means procedure to learn codebooks, applies the method sequentially to layers, and employs distillation-based fine-tuning followed by global fine-tuning to prevent error drift. The approach achieves state-of-the-art-like compression for ResNet-50 (semi-supervised) down to 5 MB with ~76% top-1 on ImageNet and extends to Mask R-CNN with substantial compression while maintaining competitive performance. This yields practical, CPU-friendly models with strong accuracy at very small memory footprints, highlighting the method’s potential for on-device deployment and future expansions to more architectures and nonlinearity-aware reconstructions.

Abstract

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.

Figures (3)

• Figure 1: Illustration of our method. We approximate a binary classifier $\varphi$ that labels images as dogs or cats by quantizing its weights. Standard method: quantizing $\varphi$ with the standard objective function \ref{['eq:pq_obj']} promotes a classifier $\widehat{\varphi}_{\text{standard}}$ that tries to approximate $\varphi$ over the entire input space and can thus perform badly for in-domain inputs. Our method: quantizing $\varphi$ with our objective function \ref{['eq:ours_obj']} promotes a classifier $\widehat{\varphi}_{\text{activations}}$ that performs well for in-domain inputs. Images lying in the hatched area of the input space are correctly classified by $\varphi_{\text{activations}}$ but incorrectly by $\varphi_{\text{standard}}$.
• Figure 2: We quantize $C_{\text{out}}$ filters of size $C_{\text{in}}\times K\times K$ using a subvector size of $d = K\times K$. In other words, we spatially quantize the convolutional filters to take advantage of the redundancy of information in the network. Similar colors denote subvectors assigned to the same codewords.
• Figure 3: Compression results for ResNet-18 and ResNet-50 architectures. We explore two compression regimes as defined in Section \ref{['subsec:setup']}: small block sizes (block sizes of $d$ = 4 and 9) and large block sizes (block sizes $d$ = 8 and 18). The results of our method for $k=256$ centroids are of practical interest as they correspond to a byte-compatible compression scheme.