Prism: Unleashing GPU Sharing for Cost-Efficient Multi-LLM Serving

TL;DR

Prism tackles the cost and latency challenges of serving many LLMs by enabling dynamic cross-model GPU memory sharing. It introduces kvcached to decouple virtual and physical memory and a two-level scheduling framework that balances memory demands across GPUs and models. The approach yields substantial improvements on real traces, including over 2x cost savings and up to 3.3x SLO attainment, and scales to large model sets with practical GPU counts. The work demonstrates a viable path toward cost-efficient, SLO-aware multi-LLM serving and provides a prototype for public release.

Abstract

Serving large language models (LLMs) is expensive, especially for providers hosting many models, making cost reduction essential. The unique workload patterns of serving multiple LLMs (i.e., multi-LLM serving) create new opportunities and challenges for this task. The long-tail popularity of models and their long idle periods present opportunities to improve utilization through GPU sharing. However, existing GPU sharing systems lack the ability to adjust their resource allocation and sharing policies at runtime, making them ineffective at meeting latency service-level objectives (SLOs) under rapidly fluctuating workloads. This paper presents Prism, a multi-LLM serving system that unleashes the full potential of GPU sharing to achieve both cost efficiency and SLO attainment. At its core, Prism tackles a key limitation of existing systems$\unicode{x2014}$the lack of $\textit{cross-model memory coordination}$, which is essential for flexibly sharing GPU memory across models under dynamic workloads. Prism achieves this with two key designs. First, it supports on-demand memory allocation by dynamically mapping physical to virtual memory pages, allowing flexible memory redistribution among models that space- and time-share a GPU. Second, it improves memory efficiency through a two-level scheduling policy that dynamically adjusts sharing strategies based on models' runtime demands. Evaluations on real-world traces show that Prism achieves more than $2\times$ cost savings and $3.3\times$ SLO attainment compared to state-of-the-art systems.

Figures (12)

• Figure 1: The workload characteristics of multi-LLM serving from a representative service provider, Hyperbolic hyperbolic.
• Figure 2: A detailed analysis of four production multi-LLM serving traces. These figures show that long-tail model popularity exists in all traces (a), many models have frequent idle period (b)+(c), and the request rate can fluctuate rapidly (d).
• Figure 3: The size of KV cache memory different GPU sharing systems can use on an example workload.
• Figure 4: Memory sizes directly impact inference performance.
• Figure 5: The system architecture and design overview of Prism.