Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

Hang Zhang, Jiuchen Shi, Yixiao Wang, Quan Chen, Yizhou Shan, Minyi Guo

TL;DR

FASTLIBRA addresses inefficiencies in Multi-LoRA serving by introducing a dependency-aware cache manager and a cost-based cache swapper that share a unified memory pool for LoRAs and KV caches. It constructs a tree-based usage dependency between LoRAs and their KV caches, ensuring valid caches remain in fast memory and coordinating swaps with a unified cost model. Empirical results across multiple LLaMA bases and application scenarios show substantial TTFT reductions (~60%), TPOT reductions (~34%), and up to ~1.7x peak throughput gains over strong baselines, with hardware-agnostic applicability. The work provides a practical framework for dynamic, efficient caching in Multi-LoRA deployments and can be integrated into existing LLM serving systems to improve consistency under varying workloads.

Abstract

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.

Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

TL;DR

FASTLIBRA addresses inefficiencies in Multi-LoRA serving by introducing a dependency-aware cache manager and a cost-based cache swapper that share a unified memory pool for LoRAs and KV caches. It constructs a tree-based usage dependency between LoRAs and their KV caches, ensuring valid caches remain in fast memory and coordinating swaps with a unified cost model. Empirical results across multiple LLaMA bases and application scenarios show substantial TTFT reductions (~60%), TPOT reductions (~34%), and up to ~1.7x peak throughput gains over strong baselines, with hardware-agnostic applicability. The work provides a practical framework for dynamic, efficient caching in Multi-LoRA deployments and can be integrated into existing LLM serving systems to improve consistency under varying workloads.

Abstract

Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.
Paper Structure (31 sections, 6 equations, 16 figures, 1 table)

This paper contains 31 sections, 6 equations, 16 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Multi-LoRA Serving with Caching
  4. Low Multi-LoRA Serving Performance
  5. Diving into Underlying Reasons
  6. Inefficient HBM Usage
  7. Inappropriate Swap-in/out of KVs and LoRAs
  8. FastLibra Methodology
  9. Challenges of FastLibra
  10. Overview of FastLibra
  11. Dependency-aware Cache Manager
  12. Usage Dependency Constructing
  13. Dependency Maintaining During Inference
  14. Implementing Unified Tree-based Caching
  15. Performance-driven Cache Swapper
  16. ...and 16 more sections

Figures (16)

  • Figure 1: An example caching state to show the Usage Dependencies between LoRAs and KV caches.
  • Figure 2: The TTFT of vLLM for various scenarios.
  • Figure 3: Examples of serving two queries under: (a) without usage dependencies, and (b) maintaining usage dependencies.
  • Figure 4: The utilization rate of the HBM space allocated to LoRAs and KV caches over time in the translation scenario.
  • Figure 5: The relationships among the visited frequency, swap costs, LRU of the KV caches or LoRAs.
  • ...and 11 more figures