GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models

TL;DR

This work tackles the memory bottleneck of KV caches in autoregressive LLM inference by introducing GPU-accelerated per-channel INT8 quantization. A CPU baseline and four CUDA kernel variants (naive, tiled, coarsened, vectorized) are evaluated across realistic workloads up to 1 billion elements, showing memory reduction by a factor of 4 with minimal impact on accuracy. The vectorized GPU kernel achieves up to $1,694×$ speedups over CPU, with maximum per-element reconstruction error bounded at $0.00394$ and mean attention error under $0.1$ even for $D=8192$, while overhead remains between $6$–$58$ ms in practical scenarios. These results indicate that quantized KV caches offer a practical, low-overhead solution to memory pressure in production LLM inference and provide concrete guidance on kernel selection and data layouts for deployment.

Abstract

The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We implement and evaluate GPU-accelerated INT8 quantization for KV cache compression, achieving 4$\times$ memory reduction with minimal accuracy degradation. We develop four CUDA kernel variants -- naive, tiled, coarsened, and vectorized -- and benchmark them across realistic workload sizes up to 1 billion elements. Our vectorized kernel achieves up to 1,694$\times$ speedup over CPU baselines while maintaining reconstruction error below 0.004 and attention score error below 0.1 even for 8K-dimensional heads. These results demonstrate that INT8 quantization provides a practical approach for reducing memory pressure in LLM inference with negligible computational overhead (6--58ms) and minimal impact on downstream model behavior

Figures (5)

• Figure 1: GPU kernel speedup comparison across all test configurations.
• Figure 2: Execution Time: CPU vs GPU
• Figure 3: GPU Performance on realistic LLM loads.
• Figure 4: L2 Reconstruction & Attention Score Error vs Matrix Size.
• Figure 5: Speed Scale Ups vs Problem Size.