ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM

TL;DR

ROMA addresses the challenge of on-device LLM inference by using a hybrid ROM-SRAM design that stores quantized base models in ROM and LoRA adapters plus KV cache in SRAM. The core idea is that a quantized base model is stable enough to reside in immutable ROM, while LoRA modules remain flexible and updatable in SRAM, enabling task-specific adaptation without altering the base weights. The authors introduce B-ROM to dramatically reduce ROM area and Fused-Cell to optimize physical layout, achieving on-chip storage for 4-bit 3B or 2-bit 8B LLaMA models with remarkable throughput (over 20k tokens/s) and low prefilling latency, outperforming CPU and GPU baselines on edge-like tasks. This approach offers a practical path to real-time, privacy-preserving LLM services on resource-constrained devices by minimizing external memory accesses and efficiently balancing mixed-precision computations.

Abstract

As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction. QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and computational costs while utilizing LoRA for task-specific adaptability. In this work, we propose ROMA, a QLoRA accelerator with a hybrid storage architecture that uses ROM for quantized base models and SRAM for LoRA weights and KV cache. Our insight is that the quantized base model is stable and converged, making it well-suited for ROM storage. Meanwhile, LoRA modules offer the flexibility to adapt to new data without requiring updates to the base model. To further reduce the area cost of ROM, we introduce a novel B-ROM design and integrate it with the compute unit to form a fused cell for efficient use of chip resources. ROMA can effectively store both a 4-bit 3B and a 2-bit 8B LLaMA model entirely on-chip, achieving a notable generation speed exceeding 20,000 tokens/s without requiring external memory.

Figures (12)

• Figure 1: QLoRA, a widely adopted technique for efficient LLM deployment on edge devices.
• Figure 2: Storage density of SRAM versus ROM.
• Figure 3: ROMA overall architecture. It is a QLoRA accelerator with hybrid ROM and SRAM.
• Figure 4: The comparisons between standard ROM and B-ROM. B-ROM need fewer CMOS transistors than ROM.
• Figure 5: Our Fused-cell layout shows better area efficiency over macro-based one.