D-Legion: A Scalable Many-Core Architecture for Accelerating Matrix Multiplication in Quantized LLMs

TL;DR

D-Legion presents a scalable many-core accelerator for quantized LLM attention workloads, built from Legions of adaptive-precision systolic array cores to accelerate matrix multiplications. It combines sparsity-aware, block-structured processing with efficient psum reduction and a flexible NoC to maximize data reuse across heads and KV sharing in GQA/MQA attention. The design space exploration identifies optimal Legion/core granularity, and a comprehensive evaluation shows up to 1.085 EOPS peak throughput with 64 Legions and substantial latency and memory savings relative to WS, DiP, ADiP, and Google TPUv4i on BitNet attention workloads. The results demonstrate significant potential for energy-efficient, high-throughput inference in quantized LLMs and establish a scalable hardware pathway for larger models and more aggressive sparsity patterns.

Abstract

The performance gains obtained by large language models (LLMs) are closely linked to their substantial computational and memory requirements. Quantized LLMs offer significant advantages with extremely quantized models, motivating the development of specialized architectures to accelerate their workloads. This paper proposes D-Legion, a novel scalable many-core architecture, designed using many adaptive-precision systolic array cores, to accelerate matrix multiplication in quantized LLMs. The proposed architecture consists of a set of Legions where each Legion has a group of adaptive-precision systolic arrays. D-Legion supports multiple computation modes, including quantized sparse and dense matrix multiplications. The block structured sparsity is exploited within a fully-sparse, or partially-sparse windows. In addition, memory accesses of partial summations (psums) are spatially reduced through parallel accumulators. Furthermore, data reuse is maximized through optimized scheduling techniques by multicasting matrix tiles across the Legions. A comprehensive design space exploration is performed in terms of Legion/core granularity to determine the optimal Legion configuration. Moreover, D-Legion is evaluated on attention workloads from two BitNet models, delivering up to 8.2$\times$ lower latency, up to 3.8$\times$ higher memory savings, and up to 3$\times$ higher psum memory savings compared to state-of-the-art work. D-Legion, with eight Legions and 64 total cores, achieves a peak throughput of 135,68 TOPS at a frequency of 1 GHz. A scaled version of D-Legion, with 32 Legions, is compared to Google TPUv4i, achieving up to 2.5$\times$ lower total latency, up to 2.3$\times$ higher total throughput, and up to 2.7$\times$ higher total memory savings.

Figures (11)

• Figure 1: Different attention layer types, including standard multi-head attention (MHA), grouped query attention (GQA), and multi-query attention (MQA).
• Figure 2: A comprehensive analysis of single large core versus many smaller cores with the same number of PEs. (a) Input bandwidths of core(s), accumulation, and psum memories versus core topology. (b) TFU for each configuration. (c) Total Number of PEs per each configuration. (d) Latency breakdown across the analyzed configurations based on attention workloads, respectively.
• Figure 3: Granularity analysis of the cores per Legion. (a) Input bandwidths of the Legion, accumulators, and psum memories versus core topology. (b) TFU per each Legion's configuration. (c) Total PEs per each Legion's configuration. (d) Latency breakdown across different Legion configurations based on attention workloads.
• Figure 4: Per-Legion configuration rate index (CRI), evaluating each Legion configuration to select the optimal number of cores and core size.
• Figure 5: (a) D-Legion architecture block diagram, consisting of $L$ Legions where each Legion has eight ADiP cores, accumulators, psum memories, Legion mapper, and local crossbar. (b) ADiP core block diagram, consisting of $D$×$D$ adaptive-precision systolic array. (c) Reconfigurable PE block diagram, consisting of 16 multipliers and arranged into four groups with their internal accumulators.