PQA: Exploring the Potential of Product Quantization in DNN Hardware Acceleration

TL;DR

This work systematically evaluates product quantization as a paradigm for DNN inference acceleration and delivers the first hardware accelerator (PQA) tailored to PQ-DNNs. By converting layer computations into memory-lookups over prototype banks and a LUT of precomputed dot-products, the authors demonstrate tangible throughput-per-area improvements on compact networks, with modest accuracy trade-offs, and explore aggressive low-bitwidth PQ regimes. The study combines algorithmic analyses, training enhancements, and a cycle-accurate FPGA-based accelerator to reveal the nuanced compute/memory/accuracy trade-offs and to quantify acceleration potential across networks and memory systems. The results show that, with carefully chosen PQ configurations and hardware co-design, PQ can outperform conventional accelerators for specific PQ-DNN settings, offering a viable path for edge-device inference under tight area and power constraints. The work also identifies practical limitations and outlines future directions including scaling to larger models, richer PQ parameterizations, and integration with complementary compression techniques.

Abstract

Conventional multiply-accumulate (MAC) operations have long dominated computation time for deep neural networks (DNNs), espcially convolutional neural networks (CNNs). Recently, product quantization (PQ) has been applied to these workloads, replacing MACs with memory lookups to pre-computed dot products. To better understand the efficiency tradeoffs of product-quantized DNNs (PQ-DNNs), we create a custom hardware accelerator to parallelize and accelerate nearest-neighbor search and dot-product lookups. Additionally, we perform an empirical study to investigate the efficiency--accuracy tradeoffs of different PQ parameterizations and training methods. We identify PQ configurations that improve performance-per-area for ResNet20 by up to 3.1$\times$, even when compared to a highly optimized conventional DNN accelerator, with similar improvements on two additional compact DNNs. When comparing to recent PQ solutions, we outperform prior work by $4\times$ in terms of performance-per-area with a 0.6% accuracy degradation. Finally, we reduce the bitwidth of PQ operations to investigate the impact on both hardware efficiency and accuracy. With only 2-6-bit precision on three compact DNNs, we were able to maintain DNN accuracy eliminating the need for DSPs.

Figures (14)

• Figure 1: Transforming a convolutional layer into its PQ equivalent. First, both input and weights tensors need to be unrolled, resulting in $\mathbf{\hat{X}}$ and $\mathbf{\hat{W}}$. The input matrix $\mathbf{\hat{X}}$ is subdivided into $N_s$ subspaces (three in this diagram), and the sub-columns in each one will be encoded using their respective bank of of prototypes $\mathbf{B}_{l}\!=\! [\mathbf{b}^{(0)}, \mathbf{b}^{(1)}, \mathbf{b}^{(2)}]$. Given a distance metric $d(\cdot)$, the input is encoded in a soft manner during training but replaced with a hard one-hot encoding during inference. For deployment, all the pre-computed dot products are stored in $\text{LUT}_{\text{PQ}}$.
• Figure 2: Speedup of PQ vs conventional convolutions on different hardware when run on all unique layers of ResNet20. CPU is 13th Gen Intel(R) Core(TM) i9-13900K and GPU is NVIDIA GeForce RTX 4090.
• Figure 3: Block diagram of PQA, the proposed custom hardware implementation for PQ.
• Figure 4: As the temperature $\tau$ decreases, the error in the output of a PQ layer increases rapidly. Even when prototypes and layer parameters are finetuned to minimise MSE($\mathbf{Y}_{\text{PQ}}$,$\mathbf{Y}$), most of the {$N_{\text{p}}$,$L_{\text{s}}$} configurations (each configuration is a dot) leads to much larger error (see rightmost subplot).
• Figure 5: Analysis of a layer with a $366\!\times\!4\!\times\!4$ input. When spatial dimensions of the input are small relative to their channel dimension, fewer prototypes lead to similar error to other configurations that result in larger $\text{LUT}_{\text{PQ}}$ and a higher $\text{FLOPs}^{\text{enc}}$ (area of grey-shaded squares) as in Eq. \ref{['eq:flops_footprint']}. The red line indicates the size in the equivalent non-PQ layer.