Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA

TL;DR

The paper tackles the inefficiency of switching multiple LoRA adapters during LLM inference by enabling cross-model KV-cache reuse between the base and activated LoRA adapters. It extends the vLLM serving engine with base-aligned block hashing and activation-aware masking to support aLoRA while preserving existing optimizations, achieving substantial end-to-end and time-to-first-token speedups in multi-turn, multi-adapter pipelines. Key contributions include the first complete realization of cross-model KV-cache reuse in modern LLM serving, quantified improvements up to 58x E2E latency reduction and 100x TTFT, and a detailed analysis of how speedups propagate through queue, prefill, and decode stages. The work bridges parameter-efficient adaptation with high-performance serving, enabling efficient, dynamic adapter activation in real-world inference stacks.

Abstract

Modern large language model (LLM) systems increasingly rely on multi-turn pipelines that are composed of multiple task-specific adapters, yet existing serving frameworks remain inefficient, incurring substantial recomputation overhead when switching between adapters. We present the first LLM serving engine that supports cross-model prefix cache reuse between base and adapted models via Activated LoRA (aLoRA), enabling efficient and fine-grained adapter switching during inference. Our design extends the vLLM framework by introducing base-aligned block hashing and activation-aware masking within the model execution path, permitting cache reuse across models while preserving compatibility with existing serving engine optimizations. Integrated into a production-grade inference stack, this approach supports dynamic adapter activation without excessive key-value tensor recomputation. Evaluation across representative multi-turn, multi-adapter pipelines demonstrates up to 58x end-to-end latency reduction and over 100x time-to-first-token improvement relative to standard LoRA baselines, with benefits that scale with model size and sequence length and manifest across all stages of the request lifecycle. This work bridges parameter-efficient model adaptation with high-performance serving, providing the first complete realization of cross-model KV-cache reuse in modern LLM inference engines.

Figures (15)

• Figure 1: Physical KV-cache memory management with and without PagedAttention, adapted from Kwon et al. kwon2023efficientmemorymanagementlarge. Example shows cache state of two requests at the same time, where $<$resv$>$ represents slots reserved based on the potential maximum sequence length of a request.
• Figure 2: vLLM system overview, adapted from Kwon et al. kwon2023efficientmemorymanagementlarge.
• Figure 3: Example of how aLoRA reuses the base model's prefix cache within our vLLM implementation. Block size is set to 3 tokens in this example. The aLoRA activation tokens are not cached as they do not constitute a full block. Adapter ID is only incorporated into the hash of tokens generated using the adapter following adapter invocation.
• Figure 4: Comparison of LoRA vs. aLoRA cache reuse in two example multi-adapter, multi-turn pipelines. Latency savings are highlighted in red. Since input cache is interchangeable across the base model and all aLoRAs finetuned from it, latency savings scale with the number of adapters and can be leveraged in multiple settings.
• Figure 5: Life cycle of an aLoRA request from initial input to final output. This figure shows one request for simplicity, but the actual aLoRA mask covers all requests in a batch simultaneously and accounts for varying points of invocation.