iServe: An Intent-based Serving System for LLMs

TL;DR

This work tackles the challenge of deploying large language models to satisfy diverse user intents without exhaustive manual configuration. It introduces iServe, an intent-based serving system that uses lightweight LLM fingerprints to quickly profile and estimate latency and memory across deployment configurations, then selects configurations and GPU placements that best meet the specified intent under current resource availability. Empirical results across multiple LLMs and Azure traces show substantial improvements in latency, cost, SLO attainment, and GPU throughput, while dramatically reducing profiling costs via fingerprint-based profiling. The approach is modular, scalable, and open-sourced, offering a practical pathway to efficient, intent-aware LLM serving in real-world clusters.

Abstract

Large Language Models (LLMs) are becoming ubiquitous across industries, where applications demand they fulfill diverse user intents. However, developers currently face the challenge of manually exploring numerous deployment configurations - combinations of parallelism and compression techniques that impact resource usage, latency, cost, and accuracy - to meet these intents. Assessing the impact of these configurations on user metrics requires extensive, costly profiling for each model. Existing approaches avoid this expense by using fixed, static configurations, but this often leads to sub-optimal performance and higher costs. Moreover, none of these solutions dynamically adapt to changing user intents to balance latency and cost, effectively. We present iServe, an automated, intent-based system for distributed LLM inference. Instead of manually selecting deployment configurations, developers simply specify their intent - such as minimizing latency, reducing cost, or meeting specific targets for either. iServe introduces fingerprints, lightweight representations of LLMs, to efficiently estimate how different configurations impact latency and memory usage. Based on these insights and GPU availability, iServe dynamically selects the optimal configuration to align with the user's intent. For various LLMs and query arrival rates, iServe best meets user intents compared to state-of-the-art systems by reducing latency by 77.62% and SLO violations by 7.09x while improving GPU throughput by 4.72x. Moreover, iServe's fingerprint-based profiling reduces profiling cost by 6.05x (GPU-hours) compared to baselines.

Figures (24)

• Figure 1: Each LLM has hundreds of deployment configurations with trade-offs in latency and memory consumption. The best configuration is specific to the user intent. See § \ref{['sec:introduction']}.
• Figure 2: Architectures of a decoder-only LLM (left) and its fingerprint (right). The fingerprint has the same components as the full LLM, but only 1-2 hidden layers. See § \ref{['sec:background']}.
• Figure 3: The best parallelism configuration (TP, PP) is user-intent and LLM-specific. See § \ref{['sec:parallelism']}.
• Figure 4: The impact of quantization on latency is LLM-specific. See § \ref{['sec:quantization']}.
• Figure 5: Pruning's effect on latency is LLM-specific. See § \ref{['sec:pruning']}.