Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Kernel Mapping and Comprehensive System Evaluation of LLM Acceleration on a CGLA

Takuto Ando, Yu Eto, Ayumu Takeuchi, Yasuhiko Nakashima

TL;DR

The paper tackles the energy inefficiency of state-of-the-art LLM inference on GPUs by evaluating a general-purpose Coarse-Grained Linear Array (CGLA) accelerator, IMAX3, for Qwen3 models using llama.cpp in an end-to-end setup. It demonstrates that a non-AI-specialized CGRA can achieve substantial energy efficiency, with PDP improvements up to 44.4x and EDP improvements up to 11.5x over a RTX 4090, while identifying host-accelerator data transfer as a critical bottleneck. The work provides a holistic assessment, including FPGA-based prototyping and 28nm ASIC projections, and offers design guidance on host interfaces and low-bit quantization co-design to enable scalable, power-constrained LLM deployment. Overall, it validates CGRA-based acceleration as a viable path for sustainable LLM inference in settings where power and thermal constraints are paramount.

Abstract

Large Language Models (LLMs) demand substantial computational resources, resulting in high energy consumption on GPUs. To address this challenge, we focus on Coarse-Grained Reconfigurable Arrays (CGRAs) as an effective alternative that provides a trade-off between energy efficiency and programmability. This paper presents the first comprehensive, end-to-end evaluation of a non-AI-specialized Coarse-Grained Linear Array (CGLA) accelerator for the state-of-the-art Qwen LLM family. The architecture has a general-purpose, task-agnostic design, yet its flexible instruction set allows for domain-specific adaptations. This flexibility enables the architecture to achieve high efficiency for sustainable LLM inference. We assess the performance of our architecture on an FPGA prototype using the widely adopted llama.cpp framework. We then project its potential as a 28nm ASIC and compare it against a high-performance GPU (NVIDIA RTX 4090) and an edge AI device (NVIDIA Jetson AGX Orin). While GPUs exhibit lower latency, our non-AI-specific accelerator achieves higher energy efficiency, improving the Power-Delay Product (PDP) by up to 44.4x and 13.6x compared with the RTX 4090 and Jetson, respectively. Similarly, it reduces the Energy-Delay Product (EDP) by up to 11.5x compared to the high-performance GPU, demonstrating a favorable performance-energy trade-off. Critically, our system-level analysis identifies host-accelerator data transfer as the primary performance bottleneck, a factor often overlooked in kernel-level studies. These findings provide design guidance for next-generation LLM accelerators. This work validates CGRAs as a suitable platform for LLM inference in power-constrained environments, without being confined to specific algorithms.

Efficient Kernel Mapping and Comprehensive System Evaluation of LLM Acceleration on a CGLA

TL;DR

The paper tackles the energy inefficiency of state-of-the-art LLM inference on GPUs by evaluating a general-purpose Coarse-Grained Linear Array (CGLA) accelerator, IMAX3, for Qwen3 models using llama.cpp in an end-to-end setup. It demonstrates that a non-AI-specialized CGRA can achieve substantial energy efficiency, with PDP improvements up to 44.4x and EDP improvements up to 11.5x over a RTX 4090, while identifying host-accelerator data transfer as a critical bottleneck. The work provides a holistic assessment, including FPGA-based prototyping and 28nm ASIC projections, and offers design guidance on host interfaces and low-bit quantization co-design to enable scalable, power-constrained LLM deployment. Overall, it validates CGRA-based acceleration as a viable path for sustainable LLM inference in settings where power and thermal constraints are paramount.

Abstract

Large Language Models (LLMs) demand substantial computational resources, resulting in high energy consumption on GPUs. To address this challenge, we focus on Coarse-Grained Reconfigurable Arrays (CGRAs) as an effective alternative that provides a trade-off between energy efficiency and programmability. This paper presents the first comprehensive, end-to-end evaluation of a non-AI-specialized Coarse-Grained Linear Array (CGLA) accelerator for the state-of-the-art Qwen LLM family. The architecture has a general-purpose, task-agnostic design, yet its flexible instruction set allows for domain-specific adaptations. This flexibility enables the architecture to achieve high efficiency for sustainable LLM inference. We assess the performance of our architecture on an FPGA prototype using the widely adopted llama.cpp framework. We then project its potential as a 28nm ASIC and compare it against a high-performance GPU (NVIDIA RTX 4090) and an edge AI device (NVIDIA Jetson AGX Orin). While GPUs exhibit lower latency, our non-AI-specific accelerator achieves higher energy efficiency, improving the Power-Delay Product (PDP) by up to 44.4x and 13.6x compared with the RTX 4090 and Jetson, respectively. Similarly, it reduces the Energy-Delay Product (EDP) by up to 11.5x compared to the high-performance GPU, demonstrating a favorable performance-energy trade-off. Critically, our system-level analysis identifies host-accelerator data transfer as the primary performance bottleneck, a factor often overlooked in kernel-level studies. These findings provide design guidance for next-generation LLM accelerators. This work validates CGRAs as a suitable platform for LLM inference in power-constrained environments, without being confined to specific algorithms.

Paper Structure

This paper contains 19 sections, 2 equations, 16 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Hardware Accelerators
  4. FPGA for LLM Inference
  5. CGRA for LLM Inference
  6. CGLA architecture and IMAX
  7. LLM Implementation on a CGLA
  8. Execution Models and Framework Overview
  9. Quantization and Supported Kernels
  10. Kernel Mapping on IMAX
  11. LMM Management and Configuration
  12. Experiments and Results
  13. Experimental Setup
  14. E2E Performance Evaluation
  15. Discussion
  16. ...and 4 more sections

Figures (16)

  • Figure 1: High-level overview of the IMAX3 system architecture, implemented on a multi-FPGA platform with four AMD Versal VPK180 devices.
  • Figure 2: Inter-PE and LMM connections within a single IMAX compute lane.
  • Figure 3: Detailed architecture of a single IMAX PE.
  • Figure 4: Overview of the LLM inference architecture, based on the llama.cpp framework, and our proposed task partitioning.
  • Figure 5: Detailed dataflow diagram for the parallelized Q8_0 dot-product operation. This kernel uses parallel SML8 and AD24 custom instructions to perform the dot-product operation.
  • ...and 11 more figures