Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems

TL;DR

This paper identifies that rank heterogeneity among LoRA adapters in multi-tenant LLM inference leads to significant performance skew and SLO violations. It introduces LoRAServe, a workload-aware, dynamic adapter placement and routing framework that distributes adapters across a cluster and uses GPUDirect RDMA to fetch adapters on demand, minimizing rank-induced interference. Through production traces and open datasets, LoRAServe demonstrates up to 2× throughput, up to 9× lower TTFT, and up to 50% fewer GPUs under SLO constraints, along with substantial reductions in adapter memory footprint. The work provides a practical, scalable solution for efficient, large-scale LoRA serving in enterprise deployments and informs future design of rank-aware cluster orchestration for PEFT workloads.

Abstract

Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems.

Figures (23)

• Figure 1: Performance per adapter when two adapters are co-served from the same Llama 7B instance. Higher rank heterogeneity leads to greater variability.
• Figure 2: A conventional LLM cluster serving LoRA workloads.
• Figure 3: Time-To-First-Token (TTFT) (top) and Time-Between-Tokens (TBT) (bottom) of different ranks against Input Size for Llama 7B.
• Figure 4: Relative TTFT of different ranks on increasing model size. Input size = 2000 and TP = 8.
• Figure 5: Relative TTFT of different ranks on Llama 7B on increasing TP. Input size = 2000.