Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compression

TL;DR

LLMs face memory bandwidth and capacity bottlenecks during inference, particularly due to the KV cache. Ecco introduces entropy-aware cache compression by fusing group-wise non-uniform quantization with pre-defined shared k-means patterns and Huffman coding, paired with a highly parallel Huffman-based decoder and runtime encoder. The approach yields up to 2.9x speedups over AWQ and 2.4x over Olive, while expanding memory capacity by ~4x and preserving state-of-the-art accuracy across multiple models and tasks. By integrating with GPU memory hierarchies and offering hardware-friendly, online compression/decompression, Ecco enables more scalable and deployable large-scale AI systems across platforms.

Abstract

Large language models (LLMs) have demonstrated transformative capabilities across diverse artificial intelligence applications, yet their deployment is hindered by substantial memory and computational demands, especially in resource-constrained environments. Quantization techniques have emerged as a critical solution, reducing data precision to enhance memory and computational efficiency. However, existing methods often suffer from high runtime overheads and potential accuracy degradation. To address these challenges, we propose Ecco, an entropy-based cache compression technique tailored for LLMs. Ecco combines group-wise and non-uniform quantization with pre-defined shared k-means patterns and Huffman coding to exploit the inherent entropy characteristics of LLM cache data. Recognizing the inefficiencies of traditional Huffman coding in terms of parallelism and latency, we introduce a novel parallel Huffman-based decoding process with a multi-stage pipeline design, reducing latency by two orders of magnitude and achieving throughput comparable to GPU L2 caches. Comprehensive evaluations demonstrate that Ecco achieves an up to 2.9$\times$ and 1.9$\times$ speedup over the state-of-the-art AWQ and SmoothQuant framework, 2.4$\times$ over the Olive accelerator, all while increasing memory capacity by nearly 4$\times$ and maintaining state-of-the-art LLM accuracy. These results underscore the effectiveness of our entropy-based cache compression in enhancing LLM performance and efficiency, paving the way for more deployable large-scale AI models.

Figures (14)

• Figure 1: Ecco Overview: A high-throughput parallel compressor and decompressor are integrated with the L2 cache to address memory bandwidth and capacity constraints in LLMs. Data is compressed prior to transmission to high-bandwidth memory (HBM) and decompressed upon retrieval for use by streaming multiprocessors (SMs).
• Figure 2: Unique value counts and bit efficiency across different compression methods.
• Figure 3: (a) Latency comparison between FP16 and Quarot; (b) Overhead incurred in Quarot's FFN.
• Figure 4: Overview of Ecco compression algorithm.
• Figure 5: Design space exploration showing the impact of shared k-means patterns numbers $(S)$ and Huffman codebooks numbers$(H)$ on LLaMA2-7B perplexity.