AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache

TL;DR

AttnCache introduces a cross-sequence attention-map caching mechanism to accelerate the prefill stage of LLM inference, exploiting the observed similarity of attention maps across semantically different inputs. It combines a lightweight Siamese feature projector, Faiss-based similarity search, and memory-mapped storage of attention maps to fetch and reuse maps during online prefill, avoiding repeated QKV computations. Across dense and MoE models, AttnCache delivers up to 1.6–2.0× end-to-end speedups on GPU and 1.2× on CPU with negligible accuracy loss, while maintaining favorable performance in encoder-only settings and showing compatibility with quantization and pruning. The approach targets prefill-only workloads (e.g., sentence embeddings, classification, QA) and lays groundwork for extending map reuse to decoding, with significant practical impact for latency-sensitive and throughput-focused deployments.

Abstract

Large Language Models (LLMs) are widely used in generative applications such as chatting, code generation, and reasoning. However, many realworld workloads such as classification, question answering, recommendation, and text embedding rely solely on the prefill stage of inference, where the model encodes input sequences without performing autoregressive decoding. In these prefill only scenarios, the self-attention computation becomes the primary performance bottleneck due to its quadratic complexity with respect to sequence length. In this paper, we observe that semantically different sentences often produce similar attention maps across layers and heads. Building on this insight, we propose AttnCache, a framework that accelerates the prefill stage of LLM inference by retrieving and reusing similar attention maps. Based on an attention map memorization database, AttnCache employs efficient caching and similarity search techniques to identify and reuse pre-cached attention maps during inference, thereby reducing the computational overhead of self-attention. Experimental results show that AttnCache achieves an average of 1.2x end-to-end and 2x attention speedup on CPU, and 1.6x end-to-end and 3x attention speedup on GPU, with negligible accuracy degradation.

Figures (6)

• Figure 1: Visualization of the attention maps in Llama-3.2-3B over two sentences, each with a length of 32. Sentence 1 is "This sentence: 'you should never do it.' means in one word:". Sentence 2 is "This sentence: 'how do you do that?' means in one word:". The plots reveals that although Sentence 1 and Sentence 2 have different meanings, their attention maps at different layers and different heads are similar.
• Figure 2: AttnCache overview. The search engine will identify the index of the sentence that produces the most similar attention maps based on the feature vector of the current input sentence and prefetch attention maps for each layer from the attention map database using the index. These fetched attention maps are stored in the attention cache and reused for the matrix multiplication calculation with value projection during the self-attention computation.
• Figure 3: The training of the feature projector. The feature projector maps input embedding of a sentence S into a feature vector. Then we train the feature projector using the attention maps-based loss function.
• Figure 4: Databases building include three steps. 1. Train the feature projector with input embeddings and attention maps; 2. Embed the input embeddings to feature vectors; 3. Store the feature vectors and attention maps to their respective databases. Both databases share the same index.
• Figure 5: Attention and end-to-end speedup of Llama-3-3B with AttnCache on MMLU STEM. By selecting an appropriate threshold, AttnCache achieves an average of 1.2$\times$ end-to-end and 2$\times$ attention speedup on CPU, and 1.6$\times$ end-to-end and 3$\times$ attention speedup on GPU, while incurring only a negligible drop in accuracy.