Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiKonv: Maximizing the Throughput of Quantized Convolution With Novel Bit-wise Management and Computation

Yao Chen, Junhao Pan, Xinheng Liu, Jinjun Xiong, Deming Chen

TL;DR

HiKonv addresses the underutilization of full-bitwidth MAC units when performing quantized convolution. It introduces a theory-backed bitwise packing framework that uses a high-bitwidth multiplier to realize multiple low-bitwidth convolutions in parallel, with guard bits and slice-based inputs. The paper derives throughput-optimization models, extends the approach to 1-D and DNN convolutions, and implements HiKonv on both general-purpose CPUs and FPGA DSPs. Empirical results show substantial speedups across 1-D and 2-D convs on CPUs (up to ~7.6x and ~3.2x) and FPGA (up to ~2.37x–2.61x throughput and DSP efficiency), highlighting practical impact and potential for hardware-software co-design in quantized DNNs.

Abstract

Quantization for CNN has shown significant progress with the intention of reducing the cost of computation and storage with low-bitwidth data representations. There are, however, no systematic studies on how an existing full-bitwidth processing unit, such as ALU in CPUs and DSP in FPGAs, can be better utilized to deliver significantly higher computation throughput for convolution under various quantized bitwidths. In this study, we propose HiKonv, a unified solution that maximizes the throughput of convolution on a given underlying processing unit with low-bitwidth quantized data inputs through novel bit-wise management and parallel computation. We establish theoretical framework and performance models using a full-bitwidth multiplier for highly parallelized low-bitwidth convolution, and demonstrate new breakthroughs for high-performance computing in this critical domain. For example, a single 32-bit processing unit in CPU can deliver 128 binarized convolution operations (multiplications and additions) and 13 4-bit convolution operations with a single multiplication instruction, and a single 27x18 multiplier in the FPGA DSP can deliver 60, 8 or 2 convolution operations with 1, 4 or 8-bit inputs in one clock cycle. We demonstrate the effectiveness of HiKonv on both CPU and FPGA. On CPU, HiKonv outperforms the baseline implementation with 1 to 8-bit inputs and provides up to 7.6x and 1.4x performance improvements for 1-D convolution, and performs 2.74x and 3.19x over the baseline implementation for 4-bit signed and unsigned data inputs for 2-D convolution. On FPGA, HiKonv solution enables a single DSP to process multiple convolutions with a shorter processing latency. For binarized input, each DSP with HiKonv is equivalent up to 76.6 LUTs. Compared to the DAC-SDC 2020 champion model, HiKonv achieves a 2.37x throughput improvement and 2.61x DSP efficiency improvement, respectively.

HiKonv: Maximizing the Throughput of Quantized Convolution With Novel Bit-wise Management and Computation

TL;DR

HiKonv addresses the underutilization of full-bitwidth MAC units when performing quantized convolution. It introduces a theory-backed bitwise packing framework that uses a high-bitwidth multiplier to realize multiple low-bitwidth convolutions in parallel, with guard bits and slice-based inputs. The paper derives throughput-optimization models, extends the approach to 1-D and DNN convolutions, and implements HiKonv on both general-purpose CPUs and FPGA DSPs. Empirical results show substantial speedups across 1-D and 2-D convs on CPUs (up to ~7.6x and ~3.2x) and FPGA (up to ~2.37x–2.61x throughput and DSP efficiency), highlighting practical impact and potential for hardware-software co-design in quantized DNNs.

Abstract

Quantization for CNN has shown significant progress with the intention of reducing the cost of computation and storage with low-bitwidth data representations. There are, however, no systematic studies on how an existing full-bitwidth processing unit, such as ALU in CPUs and DSP in FPGAs, can be better utilized to deliver significantly higher computation throughput for convolution under various quantized bitwidths. In this study, we propose HiKonv, a unified solution that maximizes the throughput of convolution on a given underlying processing unit with low-bitwidth quantized data inputs through novel bit-wise management and parallel computation. We establish theoretical framework and performance models using a full-bitwidth multiplier for highly parallelized low-bitwidth convolution, and demonstrate new breakthroughs for high-performance computing in this critical domain. For example, a single 32-bit processing unit in CPU can deliver 128 binarized convolution operations (multiplications and additions) and 13 4-bit convolution operations with a single multiplication instruction, and a single 27x18 multiplier in the FPGA DSP can deliver 60, 8 or 2 convolution operations with 1, 4 or 8-bit inputs in one clock cycle. We demonstrate the effectiveness of HiKonv on both CPU and FPGA. On CPU, HiKonv outperforms the baseline implementation with 1 to 8-bit inputs and provides up to 7.6x and 1.4x performance improvements for 1-D convolution, and performs 2.74x and 3.19x over the baseline implementation for 4-bit signed and unsigned data inputs for 2-D convolution. On FPGA, HiKonv solution enables a single DSP to process multiple convolutions with a shorter processing latency. For binarized input, each DSP with HiKonv is equivalent up to 76.6 LUTs. Compared to the DAC-SDC 2020 champion model, HiKonv achieves a 2.37x throughput improvement and 2.61x DSP efficiency improvement, respectively.
Paper Structure (32 sections, 3 theorems, 26 equations, 14 figures, 3 tables, 2 algorithms)

This paper contains 32 sections, 3 theorems, 26 equations, 14 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Optimized software library
  4. Leveraging Reconfigurable Hardware
  5. Motivation
  6. Preliminary
  7. Single Multiplier Multi-Channel Multiplication
  8. 1-D Convolution
  9. Multiplier for Convolution
  10. From Multiplication To Convolution
  11. 1-D Convolution Extension
  12. DNN Convolution Extension
  13. Maximizing Processing Throughput
  14. Sign Extension for Sign Integers
  15. Platform Oriented Implementations
  16. ...and 17 more sections

Key Result

Theorem 1

Assuming a multiplier, with given $A$ and $B$ input multiplicands constructed from the $N$-element sequence $f$ and the $K$-element sequence $g$, where $f$ and $g$ are quantized to $p$ and $q$ bits, respectively, with the guard bits $G_b$, we can obtain $N+K-1$ segments from the product $Prod =A\tim

Figures (14)

  • Figure 1: INT4 optimization on DSP48E2 xilinxint4.
  • Figure 2: Binary view of the ideal process of $Prod = A\times B$.
  • Figure 3: Computation of $F_{XN,K}$ 1-D convolution.
  • Figure 4: Throughput of processing units with different bitwidth settings.
  • Figure 5: A numerical example of a $F_{3,2}$ 1-D convolution.
  • ...and 9 more figures

Theorems & Definitions (6)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof