Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

TL;DR

A block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation are introduced.

Abstract

Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without persistence, every eviction forces a full re-prefill through the model -- 15.7 seconds per agent at 4K context. We address this by persisting each agent's KV cache to disk in 4-bit quantized format and reloading it directly into the attention layer, eliminating redundant O(n) prefill computation via direct cache restoration. The system comprises three components: a block pool providing per-agent isolated Q4 KV caches in safetensors format, a BatchQuantizedKVCache for concurrent inference over multiple agents' quantized caches, and cross-phase context injection that accumulates attention state across conversation phases without re-computation. Evaluated on three architectures (Gemma 3 12B, dense GQA, 48 layers; DeepSeek-Coder-V2-Lite 16B, MoE MLA, 27 layers; Llama 3.1 8B, dense GQA, 32 layers), cache restoration reduces time-to-first-token by up to 136x (Gemma: 22--136x at 4K--32K; DeepSeek: 11--76x at 4K--32K; Llama: 24--111x at 4K--16K; 3--10x at 1K). Q4 quantization fits 4x more agent contexts into fixed device memory than FP16. Perplexity measured with actual Q4 KV caches shows -0.7% for Gemma, +2.8% for Llama, and +3.0% for DeepSeek. Open-source at https://github.com/yshk-mxim/agent-memory

Figures (6)

• Figure 1: System architecture. Multiple agents maintain isolated KV caches in a persistent block pool. The Q4 pipeline quantizes cache data on save and operates directly on quantized tensors during attention. Disk persistence enables sub-100ms reload (warm) vs seconds of re-prefill (cold).
• Figure 2: TTFT scaling across cache states for all three models (Gemma solid, DeepSeek dashed, Llama dotted). Cold prefill scales linearly with context length. Hot and warm caches reduce TTFT by up to 136$\times$ at 32K (Gemma) and 111$\times$ at 16K (Llama), with sub-second reload up to 16K context. Llama 3.1 8B falls between Gemma and DeepSeek in cold prefill, but achieves the highest warm speedups (111$\times$ at 16K) due to its smaller 8B parameter count enabling fast cache reload relative to cold.
• Figure 3: Staggered request arrivals (4K cold context, Agent B arrives 2 s after Agent A). Total wall time is identical between sequential and batched modes. However, Agent B's own perceived TTFT (submit to first token) is worse in batched mode: 34.1 s vs 16.7 s for Gemma (2.0$\times$), 20.2 s vs 10.3 s for Llama (2.0$\times$), 6.8 s vs 3.9 s for DeepSeek (1.7$\times$). In sequential mode, B runs alone with full GPU bandwidth after A completes. In batched mode, B starts 2 s after A but shares prefill bandwidth via interleaved chunking, approximately doubling its individual TTFT.
• Figure 4: Architecture comparison. The block pool abstracts away architectural differences through ModelCacheSpec. Gemma 3 uses grouped-query attention with hybrid sliding-window layers, requiring 5D mask expansion and window-aware chunked prefill. DeepSeek uses multi-latent attention with asymmetric K/V dimensions (192 vs 128) and MoE routing, requiring larger memory budgets for intermediate tensors.
• Figure 5: Agent cache state across prisoner's dilemma phases. Permanent agents (Warden, Marco, Danny) start cold and transition to warm/hot as context accumulates via cross-phase injection. Each phase extends the cached prefix rather than re-computing. The Analyst appears only in Phase 5 (cold start). TTFT annotations show projected latency from Table \ref{['tab:ttft']} at equivalent context lengths.