LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents

TL;DR

LRAgent addresses the high KV cache and compute overhead in multi-LoRA LLM agent systems by decoupling the cache into a shared base component from the pretrained backbone and a compact, per-agent low-rank adapter cache. It proposes BaseShared and BaseLRShared to maximize cache sharing while preserving role-specific behavior, and introduces Flash-LoRA-Attention to reconstruct adapter contributions efficiently without full low-rank materialization. Empirically, BaseShared and BaseLRShared maintain accuracy close to the non-shared baseline (within about 0.7–1.5%), while delivering substantial throughput and TTFT improvements; Flash-LoRA-Attention yields additional latency and throughput benefits, bringing performance close to fully shared caching. The approach is validated on two multi-agent, multi-LoRA benchmarks (HotpotQA and ScienceQA) with two model families (LLaMA-3.1 and Ministral), and proved robust across qv and qkvo LoRA configurations, suggesting practical impact for scalable, memory-efficient multi-agent LLM systems.

Abstract

Role specialization in multi-LLM agent systems is often realized via multi-LoRA, where agents share a pretrained backbone and differ only through lightweight adapters. Despite sharing base model weights, each agent independently builds and stores its own KV cache for the same long, tool-augmented trajectories, incurring substantial memory and compute overhead. Existing KV cache sharing methods largely overlook this multi-LoRA setting. We observe that, across agents, cache differences are dominated by adapter outputs, while activations from the shared pretrained backbone remain highly similar. Based on this observation, we propose LRAgent, a KV cache sharing framework for multi-LoRA agents that decomposes the cache into a shared base component from the pretrained weights and an adapter-dependent component from LoRA weights. LRAgent reduces memory overhead by sharing the base component and storing the adapter component in its inherent low-rank form, and further reduces compute overhead, enabled by shared-$A$ multi-LoRA architectures, by also sharing the low-rank cache and avoiding redundant computations for contexts already processed by other agents. To efficiently reconstruct adapter contributions at runtime, we introduce Flash-LoRA-Attention, a kernel that reorders attention computation to avoid materializing the low-rank cache to full dimension. LRAgent achieves throughput and time-to-first-token latency close to fully shared caching, while preserving accuracy near the non-shared caching baseline across agentic question-answering benchmarks.

Figures (9)

• Figure 1: (Left) Relationship between the full cache, base cache, and adapter output. (Right) Layer-wise pairwise cosine similarity of the base and full caches, measured on the same context across three agent pairs using 128 samples of 2k tokens from the HotpotQA dataset.
• Figure 2: Agent iteration and cache accumulation for Non-Shared, BaseShared, and BaseLRShared. $\mathrm{T0}$ denotes the system prompt shared across agents, and $\mathrm{T}i$ denotes trajectory context blocks, formed by concatenating model-generated tokens and retrieved context from external sources. BaseShared shares only the base cache and maintains a separate LR cache per agent, whereas BaseLRShared shares both the base and LR caches.
• Figure 3: Diagram of base and LR cache computation with an initial context of length $L_p$ prefilled by agent $i$, followed by an additional context of length $L_c$ processed by agent $j$, under (a) BaseShared and (b) BaseLRShared. BaseShared maintains per-agent LR caches and computes the LR cache using hidden states for all context tokens not yet processed by the current agent, whereas BaseLRShared shares a single LR cache and uses hidden states only for newly appended tokens. Both methods first compute the base cache from the pretrained weights ( 1, 4). They then compute the LR cache via the LoRA down-projection ( 2, 5), and later expand it to the full dimension via the LoRA up-projection over the full sequence ( 3, 6). Efficient LR cache expansion is described in Section \ref{['sec:method_flashlora']}.
• Figure 4: System throughput (tokens per second) of BaseShared and BaseLRShared, with Flash-LoRA-Attention (FLA).
• Figure 5: Memory usage (GB) of cache sharing methods on total sequence length of 66.4k on Ministral-8B-Instruct.