H2PIPE: High throughput CNN Inference on FPGAs with High-Bandwidth Memory

TL;DR

H2PIPE extends the HPIPE layer-pipelined CNN accelerator to exploit High-Bandwidth Memory (HBM), enabling high-throughput inference for large CNNs on FPGAs. It develops a weight-offload strategy and a credit-based, latency-insensitive dataflow to balance on-chip and off-chip memory, supported by an automated compiler that assigns layers to HBM or on-chip buffers. The architecture achieves substantial throughput gains over prior FPGA CNN accelerators, including up to 19.4x on ResNet-18, 5.1x on ResNet-50, and 10.5x on VGG-16, while maintaining accuracy with 8-bit quantization. This work demonstrates practical scalability for large networks and highlights the importance of hybrid memory systems and deadlock-free weight distribution in FPGA-based CNN accelerators.

Abstract

Convolutional Neural Networks (CNNs) combine large amounts of parallelizable computation with frequent memory access. Field Programmable Gate Arrays (FPGAs) can achieve low latency and high throughput CNN inference by implementing dataflow accelerators that pipeline layer-specific hardware to implement an entire network. By implementing a different processing element for each CNN layer, these layer-pipelined accelerators can achieve high compute density, but having all layers processing in parallel requires high memory bandwidth. Traditionally this has been satisfied by storing all weights on chip, but this is infeasible for the largest CNNs, which are often those most in need of acceleration. In this work we augment a state-of-the-art dataflow accelerator (HPIPE) to leverage both High-Bandwidth Memory (HBM) and on-chip storage, enabling high performance layer-pipelined dataflow acceleration of large CNNs. Based on profiling results of HBM's latency and throughput against expected address patterns, we develop an algorithm to choose which weight buffers should be moved off chip and how deep the on-chip FIFOs to HBM should be to minimize compute unit stalling. We integrate the new hardware generation within the HPIPE domain-specific CNN compiler and demonstrate good bandwidth efficiency against theoretical limits. Compared to the best prior work we obtain speed-ups of at least 19.4x, 5.1x and 10.5x on ResNet-18, ResNet-50 and VGG-16 respectively.

Figures (6)

• Figure 1: CNN dataflow architecture overview
• Figure 2: A 4-Hi HBM Stack connected to the top-side of an FPGA through an interposer
• Figure 3: (a) Simulated and hardware measured read and write efficiency of an HBM pseudo-channel as a function of burst length and (b) Hardware measured read latency as a function of burst length when controller bandwidth is saturated
• Figure 4: (a) Weight distribution network for the bottom HBM stack (b) Weight write path to both top and bottom HBM stacks
• Figure 5: Deadlock scenario in a valid/ready latency-insensitive paradigm