Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache

TL;DR

Cachemir presents a KV-cache aware, fully homomorphic encrypted LLM inference regime that overcomes the latency barriers of processing autoregressive decoding under FHE. It introduces Interleaved Replicated Packing for efficient ct-pt vector-matrix multiplications, a novel dynamic KV-cache protocol for updating and computing QK^T and Softmax×V, and an augmented bootstrapping placement strategy tailored to KV-cache decoding. Together, these innovations yield up to ~48.8× and ~67.2× CPU speedups over prior works (MOAI, THOR) and enable sub-100-second GPU per-token generation on Llama-3-8B, with additional end-to-end accelerations observed across models and sequence lengths. The work demonstrates strong improvements in speed and scalability while maintaining acceptable accuracy, illustrating a path toward practical, privacy-preserving LLM inference.

Abstract

Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.

Figures (10)

• Figure 1: Auto-regressive decoding for generative inference of LLMs. The computation of $QK^T$ and $\mathrm{Softmax}(\cdot)\times V$ for previous tokens (in blue) can be saved with KV cache.
• Figure 2: A Transformer block in LLaMA.
• Figure 3: Overview of Cachemir.
• Figure 4: Toy example of our VMM packing method, where $\text{Rot}_r$ and $\text{Rot}_l$ denote right-rotate and left-rotate, respectively. (a) Ciphertext-plaintext vector matrix multiplication. (b) Example of inner rotation $\text{InRot}_l(2)$. (c) Direct packing requires 6 rotations and 2 multiplication depths. (d) Replicated packing requires 5 rotations and 3 multiplication depths. (e) Interleaved replicated packing requires 3 rotations and 1 multiplication depth, where masking useless slots can be fused with the next layer.
• Figure 5: Example of $QK^T$ and K Cache Maintenance. (a) $QK^T$ in decoding, where existing sequence length $n'=5$. (b) Interleaved packing for K Cache, where K Cache of $t=N/d=2$ tokens are packed in one ciphertext. (c) Computing process of $QK^T$. (d) Process of updating K cache when generating token 5.