Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs

TL;DR

This work presents a hardware/software co-design approach that extends RISC-V with custom functional units to accelerate DNNs exhibiting semi-structured and unstructured sparsity on FPGAs. It introduces three accelerators—SSSA for semi-structured sparsity, USSA for unstructured sparsity, and CSA that combines both—built around a baseline CFU-enabled RISC-V core. The key innovations are a lookahead encoding that embeds sparsity information into weight blocks, and a variable-cycle MAC that adapts to actual nonzero weights, enabling substantial speedups (up to fivefold) with modest FPGA resource overhead. The results on TinyML-like tasks show competitive speedups and small accuracy loss for INT8 quantization, supporting practical deployment on small FPGAs and indicating strong potential for hardware/software co-design in sparse DNN acceleration.

Abstract

The customizability of RISC-V makes it an attractive choice for accelerating deep neural networks (DNNs). It can be achieved through instruction set extensions and corresponding custom functional units. Yet, efficiently exploiting these opportunities requires a hardware/software co-design approach in which the DNN model, software, and hardware are designed together. In this paper, we propose novel RISC-V extensions for accelerating DNN models containing semi-structured and unstructured sparsity. While the idea of accelerating structured and unstructured pruning is not new, our novel design offers various advantages over other designs. To exploit semi-structured sparsity, we take advantage of the fine-grained (bit-level) configurability of FPGAs and suggest reserving a few bits in a block of DNN weights to encode the information about sparsity in the succeeding blocks. The proposed custom functional unit utilizes this information to skip computations. To exploit unstructured sparsity, we propose a variable cycle sequential multiply-and-accumulate unit that performs only as many multiplications as the non-zero weights. Our implementation of unstructured and semi-structured pruning accelerators can provide speedups of up to a factor of 3 and 4, respectively. We then propose a combined design that can accelerate both types of sparsities, providing speedups of up to a factor of 5. Our designs consume a small amount of additional FPGA resources such that the resulting co-designs enable the acceleration of DNNs even on small FPGAs. We benchmark our designs on standard TinyML applications such as keyword spotting, image classification, and person detection.

Figures (10)

• Figure 1: Different sparsity structures: (a) Structured sparsity, resulting from structured pruning, which removes whole columns or rows from, e.g., a convolution matrix. (b) Unstructured sparsity, resulting from unstructured pruning, which removes arbitrary weights. (c) Semi-structured sparsity, resutling from semi-structured pruning, a.k.a. $n$:$m$ pruning that zero-outs $n$ weights every $m$ elements; shown is a 2:4 pattern.
• Figure 2: Overview of our method for hardware/software co-design of RISC-V extensions to accelerate sparse DNN on FPGAs. The process starts with a DNN model, which is pruned using unstructured or semi-structured pruning. Software customization of DNN kernels and hardware specialization of RISC-V extensions are jointly performed according to the co-design approach.
• Figure 3: CPU-CFU interface using R-type instruction of RISC-V.
• Figure 4: RTL diagram of the proposed hardware SSSA for exploiting semi-structured sparsity.
• Figure 5: The first row shows 7 blocks of DNN weights, each containing four INT8 weights. \ref{['alg:encoding']} (encoding) performs a pass over all blocks, annotates each non-zero block with the information of the number of succeeding all-zero blocks, and encodes their number. The 2nd row shows the calculated code.