RoMe: Row Granularity Access Memory System for Large Language Models

TL;DR

The paper tackles the inefficiency of cache-line granularity in DRAM for large language models by introducing RoMe, a row-granularity memory system. RoMe replaces column-level interfaces with RD_row/WR_row, introduces a virtual bank to remove bank groups and pseudo channels, and adds a command generator to translate row commands into conventional DRAM sequences, enabling extra channels with minimal hardware. Evaluations across representative LLMs show RoMe achieving roughly 12.5% higher memory bandwidth and 9–10% reductions in time-per-output-token, with only modest area and energy overhead. This work demonstrates that row-granularity memory can simplify memory controller design and improve scalability for next-generation HBM-based AI accelerators, particularly for sequential, bulk data movement typical of LLM workloads.

Abstract

Modern HBM-based memory systems have evolved over generations while retaining cache line granularity accesses. Preserving this fine granularity necessitated the introduction of bank groups and pseudo channels. These structures expand timing parameters and control overhead, significantly increasing memory controller scheduling complexity. Large language models (LLMs) now dominate deep learning workloads, streaming contiguous data blocks ranging from several kilobytes to megabytes per operation. In a conventional HBM-based memory system, these transfers are fragmented into hundreds of 32B cache line transactions. This forces the memory controller to employ unnecessarily intricate scheduling, leading to growing inefficiency. To address this problem, we propose RoMe. RoMe accesses DRAM at row granularity and removes columns, bank groups, and pseudo channels from the memory interface. This design simplifies memory scheduling, thereby requiring fewer pins per channel. The freed pins are aggregated to form additional channels, increasing overall bandwidth by 12.5% with minimal extra pins. RoMe demonstrates how memory scheduling logic can be significantly simplified for representative LLM workloads, and presents an alternative approach for next-generation HBM-based memory systems achieving increased bandwidth with minimal hardware overhead.

Figures (14)

• Figure 1: Distribution of weight, activation, and KV cache size of DeepSeek-V3 arxiv-2024-deepseek-v3, Grok 1 github-2024-grok1, and Llama 3 arxiv-2024-llama3 in the $\texttt{prefill}$ and $\texttt{decode}$ stages.
• Figure 2: (a) Trends in data rate, core frequency, and channel width, and (b) growth of C/A pin overhead across HBM generations.
• Figure 3: Overview of HBM architecture and internal organization.
• Figure 4: Conventional memory controller architecture.
• Figure 5: Transformer-based LLM architecture.