BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching

TL;DR

BlendServe is presented, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree.

Abstract

Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular for latency-insensitive applications. Meanwhile, recent progress in model capability and modality makes requests more diverse in compute and memory demands, creating unique opportunities for throughput improvement by resource overlapping. However, a request schedule that maximizes resource overlapping can conflict with the schedule that maximizes prefix sharing, a widely-used performance optimization, causing sub-optimal inference throughput. We present BlendServe, a system that maximizes resource utilization of offline batch inference by combining the benefits of resource overlapping and prefix sharing using a resource-aware prefix tree. BlendServe exploits the relaxed latency requirements in offline batch inference to reorder and overlap requests with varied resource demands while ensuring high prefix sharing. We evaluate BlendServe on a variety of synthetic multi-modal workloads and show that it provides up to $1.44\times$ throughput boost compared to widely-used industry standards, vLLM and SGLang.

Figures (12)

• Figure 1: Three paradigms for scheduling compute- and memory-intensive requests in offline batch inference. For simplicity, the utilization of GPU tensor cores and HBM bandwidth across different operators within a batch is combined into one rectangle.
• Figure 2: Request input/output length distribution from $6$ well-known open-sourced traces, including chatbot ShareGPT, WildChat, and BurstGPT zhao2024wildchat1mchatgptinteractionzheng2024lmsyschat1mlargescalerealworldllmwang2024burstgpt, video datasets OpenVid nan2024openvid1mlargescalehighqualitydataset, benchmark traces MMLU hendrycks2021measuringmassivemultitasklanguage and production traces Azure-Trace stojkovic2024dynamollmdesigningllminference. Requests from different traces demonstrate distinct length distributions.
• Figure 3: Compute density of requests with different input/output lengths (Llama-3-70B on 8xA100 80GB GPUs). Compute density is calculated as compute usage divided by memory usage, following the definition in § \ref{['sec:request-resource-imbalance']}. The results show that requests with different output length leads to diverse resource demands.
• Figure 4: Overview of BlendServe's design. Leaf nodes in the prefix tree are actual requests while others represent the common prefix in prompts. The color of nodes represents the performance characterization of all requests within the sub-tree, which is more compute-intensive at red and memory-intensive at blue. Give a set of requests, preprocess ahead of GPU running is performed, which consists of prefix tree construction, output length sampling, and transformation including tree sorting and node splitting (➀,➁, and ➂). Then dual scanner forms the runtime batch from most compute- and memory-intensive nodes, which is consumed by the backend engine (➃). Preprocess consumes less than $1$% time compared to the end-to-end process, which is negligible.
• Figure 5: Example of memory partition on Llama-3.1-8B with an A100 80GB. The left node has requests with $512$ input length and $256$ output length which lead to a compute density of $3.73$, while the right node is memory-intensive with $256$ input length and $16384$ output length. The dual scanner will reserve $20$GB for model weights and temporary buffers, then partition the rest of memory according to the compute density.