Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling

TL;DR

The paper addresses the energy and performance implications of on-chip SRAM sizing and frequency in LLM inference by separating the compute-bound prefill phase from the memory-bound decode phase. It employs a combined simulation workflow using OpenRAM for SRAM energy, LLMCompass for matmul latency, and ScaleSIM for systolic-array utilization to evaluate configurations with varying local buffers, frequencies, and bandwidths. Key findings show that SRAM leakage largely drives total energy, with diminishing returns on latency improvements beyond ~32–64 KB of local SRAM, and that memory bandwidth caps decode performance, making bandwidth upgrades beneficial but with diminishing gains. The study provides architectural guidance for energy-efficient LLM accelerators, suggesting small local SRAM paired with high compute frequency and a focus on memory-bandwidth-aware decode optimization for datacenter deployments.

Abstract

Energy consumption dictates the cost and environmental impact of deploying Large Language Models. This paper investigates the impact of on-chip SRAM size and operating frequency on the energy efficiency and performance of LLM inference, focusing on the distinct behaviors of the compute-bound prefill and memory-bound decode phases. Our simulation methodology combines OpenRAM for energy modeling, LLMCompass for latency simulation, and ScaleSIM for systolic array operational intensity. Our findings show that total energy use is predominantly determined by SRAM size in both phases, with larger buffers significantly increasing static energy due to leakage, which is not offset by corresponding latency benefits. We quantitatively explore the memory-bandwidth bottleneck, demonstrating that while high operating frequencies reduce prefill latency, their positive impact on memory-bound decode latency is capped by the external memory bandwidth. Counter-intuitively, high compute frequency can reduce total energy by reducing execution time and consequently decreasing static energy consumption more than the resulting dynamic power increase. We identify an optimal hardware configuration for the simulated workload: high operating frequencies (1200MHz-1400MHz) and a small local buffer size of 32KB to 64KB. This combination achieves the best energy-delay product, balancing low latency with high energy efficiency. Furthermore, we demonstrate how memory bandwidth acts as a performance ceiling, and that increasing compute frequency only yields performance gains up to the point where the workload becomes memory-bound. This analysis provides concrete architectural insights for designing energy-efficient LLM accelerators, especially for datacenters aiming to minimize their energy overhead.

Figures (16)

• Figure 1: Modeled architecture: We assume a generic accelerator with a global buffer that inrefaces to memory and local buffers that serve the systolic arrays.
• Figure 2: LLMCompass computation and data transfer taken into account. Omitted layers contribute a negligible amount to either computation or data tranfer.
• Figure 3: Physical design of the 16$\times16$ systolic array with routing-wire density
• Figure 4: Prefill Cycle Count as a function of frequency (minor effect) and 1st level SRAM buffer (major effect)
• Figure 5: Decode Cycle Count as a function of frequency (major effect) and 1st level SRAM buffer (minor effect)