FPGA Co-Design for Efficient N:M Sparse and Quantized Model Inference

TL;DR

This work tackles the challenge of deploying large language models in resource-limited environments by proposing an FPGA–software co-design that unifies offline $N:M$ structured pruning with $B$-bit quantization (specifically $B=4$ for weights) and online dequantization for GEMM across CPUs, GPUs, and an FPGA accelerator. The framework supports generalized $N:M$ sparsity beyond the fixed $2:4$ pattern and leverages a zero-skipping, systolic-array FPGA design to reduce weight storage, memory traffic, and arithmetic cost while maintaining accuracy. Key contributions include an end-to-end compression and execution pipeline, a detailed FPGA architecture with zero-skipping PR, and empirical and predicted results showing up to $4\times$ weight-footprint reduction, $1.71\times$ GEMM speedup, and $1.29\times$ end-to-end latency reduction on synthetic workloads, plus a $1.36\times$ throughput-per-token improvement for LLaMA-7B under structured sparsity. The results demonstrate a practical, flexible path to deploy efficient LLM inference on edge and datacenter platforms, and the FPGA path supports broader sparsity patterns beyond current fixed hardware constraints.

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of language processing tasks. However, this success comes at the cost of substantial computation and memory requirements, which significantly impedes their deployment in resource-constrained environments. To address this challenge, this work introduces an automation framework that leverages weight pruning and low-bit quantization, and presents a hardware-software co-design method that generates accelerators on the Field-Programmable Gate Array (FPGA) platform. In particular, we implement a unified pipeline that applies N:M structured pruning and 4-bit integer quantization to reduce the memory footprint, followed by optimized dequantization and matrix multiplication to enhance LLM inference on several hardware platforms, including CPUs, NVIDIA GPUs with Dense and 2:4 Sparse Tensor Cores, and a custom systolic-array-based FPGA accelerator. Utilizing 2:4 sparsity combined with quantization on $4096 \times 4096$ matrices, our approach achieves a reduction of up to $4\times$ in weight storage and a $1.71\times$ speedup in matrix multiplication, yielding a $1.29\times$ end-to-end latency reduction compared to dense GPU baselines. Scaling analysis on the LLaMA-7B model further shows that structured sparsity enhances the throughput per token by $1.36\times$. These results demonstrate the synergy of fine-grained N:M sparsity and quantization for enabling efficient and deployable LLM inference, while the proposed FPGA accelerator offers a flexible architectural path for supporting a broader class of sparsity patterns beyond the fixed 2:4 hardware constraints.

Figures (4)

• Figure 1: Overview of the proposed pipeline. The green panel shows the offline preparation (N:M pruning, quantization, and packing). The red panel shows the benchmarking path (loading, dequantization, and matrix multiplication) across platforms. Purple cells indicate Half-Precision 16-bit Floating Point (FP16) values, white cells indicate zeros, and blue cells indicate 4-byte integer (INT4) nonzeros.
• Figure 2:
• Figure 3: Top-level dataflow of the output-stationary systolic array, showing on-chip buffers and FIFOs that stream tiles of matrix $A$ and matrix $B$ into the PE grid and collect the resulting outputs.
• Figure 4: Microarchitecture of a single PE with zero-skipping. Incoming activations (Data) and weights are first latched in input registers (triangular blocks), then passed to a comparator that checks whether the weight is zero. If the weight is nonzero, the PE performs a MAC operation and updates the partial-sum register; otherwise, the multiplier is bypassed and only the registered operands are forwarded to neighboring PEs.