KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models

TL;DR

KV-CAR addresses the KV-cache memory bottleneck in autoregressive LLM inference by introducing per-layer autoencoder compression of K and V along the embedding dimension and a similarity-driven KV-head reuse mechanism across adjacent layers. The approach is architecture-agnostic and trained in two stages to preserve downstream fidelity. Empirical results on GPT-2 and TinyLLaMA across multiple benchmarks show up to 47.85% KV-cache reduction with minimal perplexity/accuracy loss, and system-level tests on an NVIDIA A40 confirm longer contexts and bigger batches become feasible. The combination of learned KV compression and inter-layer reuse provides a practical path to memory-efficient inference that can be integrated with quantization and other memory-saving techniques.

Abstract

As Large Language Models (LLMs) scale in size and context length, the memory requirements of the key value (KV) cache have emerged as a major bottleneck during autoregressive decoding. The KV cache grows with sequence length and embedding dimension, often exceeding the memory footprint of the model itself and limiting achievable batch sizes and context windows. To address this challenge, we present KV CAR, a unified and architecture agnostic framework that significantly reduces KV cache storage while maintaining model fidelity. KV CAR combines two complementary techniques. First, a lightweight autoencoder learns compact representations of key and value tensors along the embedding dimension, compressing them before they are stored in the KV cache and restoring them upon retrieval. Second, a similarity driven reuse mechanism identifies opportunities to reuse KV tensors of specific attention heads across adjacent layers. Together, these methods reduce the dimensional and structural redundancy in KV tensors without requiring changes to the transformer architecture. Evaluations on GPT 2 and TinyLLaMA models across Wikitext, C4, PIQA, and Winogrande datasets demonstrate that KV CAR achieves up to 47.85 percent KV cache memory reduction with minimal impact on perplexity and zero shot accuracy. System level measurements on an NVIDIA A40 GPU show that the reduced KV footprint directly translates into longer sequence lengths and larger batch sizes during inference. These results highlight the effectiveness of KV CAR in enabling memory efficient LLM inference.

Figures (3)

• Figure 1: Block diagram of the KV-CAR framework. For each transformer layer, key and value representations are compressed using a learned autoencoder before insertion into the KV cache. Additionally, structurally redundant attention heads are identified and reused across adjacent layers, jointly minimizing KV-cache storage requirements while preserving inference fidelity.
• Figure 2: System-level analysis of maximum achievable sequence length as a function of batch size for GPT-2 on an NVIDIA A40 GPU under different KV-cache compression levels. Higher compression rates enable either longer context lengths at fixed batch size or larger batch sizes at fixed sequence length before running out of GPU memory.
• Figure 3: System-level analysis of maximum achievable sequence length as a function of batch size for Tiny Llama on an NVIDIA A40 GPU under different KV-cache compression levels. Higher compression rates enable either longer context lengths at fixed batch size or larger batch sizes at fixed sequence length before running out of GPU memory.