Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Acceleration

Endri Taka, Ning-Chi Huang, Chi-Chih Chang, Kai-Chiang Wu, Aman Arora, Diana Marculescu

TL;DR

The paper introduces systolic sparse tensor (SST) slices for FPGAs, enabling flexible structured sparsity (dense, 2:4, 1:3, 1:4) in in-fabric blocks to accelerate sparse and dense DNNs. It presents a 2D systolic dataflow with dedicated vertical routing, a sparse processing element, and dynamic sparse configuration to map GEMM workloads efficiently across SST slices. Empirical results show up to 5× FPGA frequency, up to 10.9× area reduction, and up to 3.52× speedups on sparse ViT/CNN models, with manageable area overhead and robust performance on non-AI benchmarks. The work demonstrates a practical path toward highly efficient, reconfigurable in-FPGA sparsity accelerators and outlines future work on dynamic sparsity and broader GEMV support.

Abstract

FPGA architectures have recently been enhanced to meet the substantial computational demands of modern deep neural networks (DNNs). To this end, both FPGA vendors and academic researchers have proposed in-fabric blocks that perform efficient tensor computations. However, these blocks are primarily optimized for dense computation, while most DNNs exhibit sparsity. To address this limitation, we propose incorporating structured sparsity support into FPGA architectures. We architect 2D systolic in-fabric blocks, named systolic sparse tensor (SST) slices, that support multiple degrees of sparsity to efficiently accelerate a wide variety of DNNs. SSTs support dense operation, 2:4 (50%) and 1:4 (75%) sparsity, as well as a new 1:3 (66.7%) sparsity level to further increase flexibility. When demonstrating on general matrix multiplication (GEMM) accelerators, which are the heart of most current DNN accelerators, our sparse SST-based designs attain up to 5x higher FPGA frequency and 10.9x lower area, compared to traditional FPGAs. Moreover, evaluation of the proposed SSTs on state-of-the-art sparse ViT and CNN models exhibits up to 3.52x speedup with minimal area increase of up to 13.3%, compared to dense in-fabric acceleration.

Systolic Sparse Tensor Slices: FPGA Building Blocks for Sparse and Dense AI Acceleration

TL;DR

The paper introduces systolic sparse tensor (SST) slices for FPGAs, enabling flexible structured sparsity (dense, 2:4, 1:3, 1:4) in in-fabric blocks to accelerate sparse and dense DNNs. It presents a 2D systolic dataflow with dedicated vertical routing, a sparse processing element, and dynamic sparse configuration to map GEMM workloads efficiently across SST slices. Empirical results show up to 5× FPGA frequency, up to 10.9× area reduction, and up to 3.52× speedups on sparse ViT/CNN models, with manageable area overhead and robust performance on non-AI benchmarks. The work demonstrates a practical path toward highly efficient, reconfigurable in-FPGA sparsity accelerators and outlines future work on dynamic sparsity and broader GEMV support.

Abstract

FPGA architectures have recently been enhanced to meet the substantial computational demands of modern deep neural networks (DNNs). To this end, both FPGA vendors and academic researchers have proposed in-fabric blocks that perform efficient tensor computations. However, these blocks are primarily optimized for dense computation, while most DNNs exhibit sparsity. To address this limitation, we propose incorporating structured sparsity support into FPGA architectures. We architect 2D systolic in-fabric blocks, named systolic sparse tensor (SST) slices, that support multiple degrees of sparsity to efficiently accelerate a wide variety of DNNs. SSTs support dense operation, 2:4 (50%) and 1:4 (75%) sparsity, as well as a new 1:3 (66.7%) sparsity level to further increase flexibility. When demonstrating on general matrix multiplication (GEMM) accelerators, which are the heart of most current DNN accelerators, our sparse SST-based designs attain up to 5x higher FPGA frequency and 10.9x lower area, compared to traditional FPGAs. Moreover, evaluation of the proposed SSTs on state-of-the-art sparse ViT and CNN models exhibits up to 3.52x speedup with minimal area increase of up to 13.3%, compared to dense in-fabric acceleration.

Paper Structure

This paper contains 27 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Architecture & Design Overview
  4. Fine-Grained Structured Sparsity
  5. Systolic Sparse Tensor Slices Architecture
  6. SST Operation
  7. Global Routing Interface & Dedicated Wires
  8. Sparse Processing Element
  9. GEMM Design Utilizing Multiple SST Slices
  10. Dense Implementation
  11. Dynamic Sparse Configuration
  12. Evaluation
  13. FPGA Architecture
  14. SST Slices Implementation
  15. Wirelength Gains Utilizing Dedicated Wires
  16. ...and 12 more sections

Figures (9)

  • Figure 1: 50% unstructured sparse matrix (a) and 2:4 (50%) structured sparse matrix along with compressed format (b).
  • Figure 2: Systolic Sparse Tensor slice architecture.
  • Figure 3: Sparsity modes in Systolic Sparse Elements of the SST slices (multiplexing logic omitted for clarity).
  • Figure 4: 2D systolic GEMM design: dense implementation (a) and dynamic configuration of all supported sparsity modes (b).
  • Figure 5: Routing wirelength comparison of dense GEMM mapped to SST and SDT_GIO slices, for various SA sizes.
  • ...and 4 more figures