Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management

Liu Qianli, Hong Zicong, Chen Fahao, Li Peng, Guo Song

TL;DR

Mell addresses the memory bottleneck of KV caches in serving large language models by enabling adaptive cross-GPU KV-cache migration and online scheduling to reduce the number of GPUs required while maintaining throughput. It introduces a boundary-aware adaptive migration mechanism and a competitive online KV-cache scheduling algorithm that partitions requests by size, uses hybrid migrations, and batches operations to limit overhead. Theoretical analysis proves a competitive ratio of at most $4/3$ relative to the optimum, and experiments with LLaMA-2 and real-world workload traces show substantial GPU-count reductions and increased GPU utilization, outperforming baseline approaches. This approach promises practical, scalable LLM serving on commodity GPU clusters by efficiently balancing computation, memory, and inter-GPU communication."

Abstract

Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems.

Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management

TL;DR

Mell addresses the memory bottleneck of KV caches in serving large language models by enabling adaptive cross-GPU KV-cache migration and online scheduling to reduce the number of GPUs required while maintaining throughput. It introduces a boundary-aware adaptive migration mechanism and a competitive online KV-cache scheduling algorithm that partitions requests by size, uses hybrid migrations, and batches operations to limit overhead. Theoretical analysis proves a competitive ratio of at most relative to the optimum, and experiments with LLaMA-2 and real-world workload traces show substantial GPU-count reductions and increased GPU utilization, outperforming baseline approaches. This approach promises practical, scalable LLM serving on commodity GPU clusters by efficiently balancing computation, memory, and inter-GPU communication."

Abstract

Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems.
Paper Structure (19 sections, 5 theorems, 4 equations, 15 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 5 theorems, 4 equations, 15 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Motivation
  4. System Overview
  5. Adaptive Request Migration
  6. Online KV Cache Scheduling
  7. System Model
  8. GPU Cluster
  9. Requests
  10. Serving Strategy
  11. System Cost
  12. Problem Formulation
  13. Online Algorithm Design
  14. Analysis
  15. Experiment
  16. ...and 4 more sections

Key Result

Theorem 1

Given any set of LLM requests $I$, the allocation obtained by our algorithm satisfies all the following properties (with a constant number of exceptions):

Figures (15)

  • Figure 1: Serving procedure of an LLM request.
  • Figure 2: The memory cost of processing a request with $4096$ tokens on LLaMA-13B.
  • Figure 3: Throughput and per-token decoding latency of serving LLaMA-13B in a prompt length of $100$ as batch size increases.
  • Figure 4: The distribution of prompt and response length.
  • Figure 5: The distribution of response lengths under various prompt lengths in Vicunna-13B.
  • ...and 10 more figures

Theorems & Definitions (10)

  • Theorem 1
  • proof
  • Theorem 2
  • Lemma 2.1
  • proof
  • Lemma 2.2
  • proof
  • proof : Proof of Theorem \ref{['competitive-ratio']}
  • Theorem 3
  • proof