How to keep pushing ML accelerator performance? Know your rooflines!
Marian Verhelst, Luca Benini, Naveen Verma
TL;DR
This paper addresses the challenge of sustaining ML accelerator performance as model size and data movement grow. It extends the roofline model to two objectives—throughput and energy—by incorporating arithmetic intensity across memory levels, enabling a unified view of compute and memory bottlenecks. The key contributions include an enhanced dual-roofline framework, analysis of dominant energy- and compute-contributing factors, and a survey of techniques (parallelism, data reuse, quantization, sparsity, near-/in-memory compute, and IMC) that shift workloads toward or along the rooflines. It also discusses design-space exploration tools and practical trade-offs between compute and memory resources, guiding future accelerator architectures toward higher utilization and broader workload flexibility. The work’s significance lies in providing a structured lens to navigate architectural choices (e.g., IMC/NMC, quantization, sparsity) and to anticipate open challenges in scale-out memory systems and programmable yet efficient accelerators.
Abstract
The rapidly growing importance of Machine Learning (ML) applications, coupled with their ever-increasing model size and inference energy footprint, has created a strong need for specialized ML hardware architectures. Numerous ML accelerators have been explored and implemented, primarily to increase task-level throughput per unit area and reduce task-level energy consumption. This paper surveys key trends toward these objectives for more efficient ML accelerators and provides a unifying framework to understand how compute and memory technologies/architectures interact to enhance system-level efficiency and performance. To achieve this, the paper introduces an enhanced version of the roofline model and applies it to ML accelerators as an effective tool for understanding where various execution regimes fall within roofline bounds and how to maximize performance and efficiency under the rooline. Key concepts are illustrated with examples from state-of-the-art designs, with a view towards open research opportunities to further advance accelerator performance.