Transitive Array: An Efficient GEMM Accelerator with Result Reuse

TL;DR

Transitive Sparsity introduces a reuse-driven optimization for GEMM that leverages previously computed results to dramatically reduce computations in quantized DNNs and LLMs. The Transitive Array is a multiplication-free accelerator using a Hasse-graph-based Scoreboard and tiling to exploit TranSparsity, including dynamic support for Attention operations. Empirical results show approximately 7.46× and 3.97× speedups with 2.31× and 1.65× energy reductions against Olive and BitVert on LLaMA models, while preserving accuracy, and significant improvements on ResNet-18 with ImageNet. This work offers a generalized, scalable approach to GEMM acceleration under quantization, enabling efficient, attention-capable inference with broad applicability across SOTA quantization techniques.

Abstract

Deep Neural Networks (DNNs) and Large Language Models (LLMs) have revolutionized artificial intelligence, yet their deployment faces significant memory and computational challenges, especially in resource-constrained environments. Quantization techniques have mitigated some of these issues by reducing data precision, primarily focusing on General Matrix Multiplication (GEMM). This study introduces a novel sparsity paradigm, transitive sparsity, which leverages the reuse of previously computed results to substantially minimize computational overhead in GEMM operations. By representing transitive relations using a directed acyclic graph, we develop an efficient strategy for determining optimal execution orders, thereby overcoming inherent challenges related to execution dependencies and parallelism. Building on this foundation, we present the Transitive Array, a multiplication-free accelerator designed to exploit transitive sparsity in GEMM. Our architecture effectively balances computational workloads across multiple parallel lanes, ensuring high efficiency and optimal resource utilization. Comprehensive evaluations demonstrate that the Transitive Array achieves approximately 7.46$\times$ and 3.97$\times$ speedup and 2.31$\times$ and 1.65$\times$ energy reduction compared to state-of-the-art accelerators such as Olive and BitVert while maintaining comparable model accuracy on LLaMA models.

Figures (16)

• Figure 1: Comparison of bit sparsity and transitive sparsity (Ours) on the binary general matrix multiplication.
• Figure 2: Quantization and bit-slicing. A weight tensor (FP16) is quantized into an Int4 weight matrix ($4 \times 4$). Bit-slicing decomposes each bit level of the Int4 matrix and reorganizes them into a binary weight matrix of shape ($16 \times 4$).
• Figure 3: Motivation of Transitive GEMM.
• Figure 4: Hasse graph and definition of prefix, suffix, and distance.
• Figure 5: Scoreboarding process. Step : Sort TransRows by Hamming order. Step : Record present TransRows in the Hasse graph. Step (Forward pass): Assign candidate prefixes to all nodes. Step (Backward pass): Propagate suffix requests for nodes without a Distance $=1$ prefix, searching for the prefix with the shortest distance and retaining the closest prefix for all nodes. Step : Generate a balanced forest. Step : Output the final Scoreboard information (SI).