Virgo: Cluster-level Matrix Unit Integration in GPUs for Scalability and Energy Efficiency

TL;DR

Virgo addresses the scalability-energy trade-offs of tightly coupled matrix units in GPUs by disaggregating a cluster-level matrix unit from the SIMT cores. It introduces a Gemmini-based matrix unit, a two-dimensional banked shared memory, a dedicated accumulator SRAM, a cluster-wide DMA, and a lightweight synchronization mechanism, all accessible via a memory-mapped asynchronous API. The approach yields substantial improvements, including up to 67.3% active-power reduction and 80.3% energy savings over Ampere-style configurations for GEMM, and strong data reuse and MAC utilization improvements across GEMM and FlashAttention-3 workloads. This cluster-level disaggregation enables larger tile sizes, better data reuse, and concurrent execution, offering a practical path to scalable, energy-efficient DNN acceleration in future GPUs.

Abstract

Modern GPUs incorporate specialized matrix units such as Tensor Cores to accelerate GEMM operations, which are central to deep learning workloads. However, existing matrix unit designs are tightly coupled to the SIMT core, restricting operation size due to register file capacity and bandwidth constraints. Such a limitation in scalability makes it difficult to simultaneously improve compute throughput and energy efficiency in GPUs. To address this challenge, we propose Virgo, a GPU microarchitecture that integrates dedicated matrix units at the SIMT core cluster level. By decoupling the matrix unit from the SIMT core, Virgo eliminates scalability constraints imposed by the core microarchitecture. Consequently, Virgo increases operation granularity at the hardware level, reducing energy overhead from core instruction processing. Physical disaggregation also enables a unified matrix unit design and offloading both operand and accumulator accesses from the register file, improving data reuse and energy efficiency. Furthermore, this disaggregation supports efficient concurrent execution of the SIMT core and matrix unit, optimizing mapping for fused DNN workloads. Our evaluations using synthesizable RTL demonstrate that Virgo achieves 67.3% and 24.2% reduction in on-chip active power consumption, compared to the baseline Ampere-style and Hopper-style core-coupled designs.

Figures (12)

• Figure 1: Overview of (a) today's GPU architecture with tightly coupled integration of Tensor Cores (TC), compared to (b) Virgo's cluster-level integration of matrix units.
• Figure 2: A high level overview of the Virgo microarchitecture. The Gemmini-based matrix unit is disaggregated out from the SIMT cores into a separate unit in the cluster. Dashed lines indicate optionally instantiated modules for evaluation, such as the core-coupled Tensor Core.
• Figure 3: The Virgo shared memory system.
• Figure 4: Execution in the SIMT core, matrix unit, and DMA during a GEMM operation on Virgo and Tensor Core-based designs.
• Figure 5: FlashAttention-3 mapping to Virgo.