PipeBoost: Resilient Pipelined Architecture for Fast Serverless LLM Scaling

TL;DR

PipeBoost addresses the high cold-start latency of serverless LLM serving on multi-GPU clusters by exploiting base-model sharing across GPUs and introducing fault-tolerant pipeline parallelism across loading and inference. It reorders and pipelines model loading, merges LoRA adapters during inference, and uses an epoch-based adapter switching mechanism to handle bursty traffic, while providing rapid recovery from GPU failures. Empirical results show substantial reductions in time to first token compared with state-of-the-art systems, including sub-second cold-start latency for OPT-1.3B, and strong fault-recovery performance, highlighting improvements in startup latency, scalability, and reliability for multi-GPU serverless deployments.

Abstract

This paper presents PipeBoost, a low-latency LLM serving system for multi-GPU (serverless) clusters, which can rapidly launch inference services in response to bursty requests without preemptively over-provisioning GPUs. Many LLM inference tasks rely on the same base model (e.g., LoRA). To leverage this, PipeBoost introduces fault-tolerant pipeline parallelism across both model loading and inference stages. This approach maximizes aggregate PCIe bandwidth and parallel computation across GPUs, enabling faster generation of the first token. PipeBoost also introduces recovery techniques that enable uninterrupted inference services by utilizing the shared advantages of multiple GPUs. Experimental results show that, compared to state-of-the-art low-latency LLM serving systems, PipeBoost reduces inference latency by 31% to 49.8%. For certain models (e.g., OPT-1.3B), PipeBoost achieves cold-start latencies in the range of a few hundred microseconds.

Figures (17)

• Figure 1: Overhead of LLM model loading.
• Figure 2: Different methods of LLM model loading on multiple GPUs. Transformers and ServerlessLLM require the entire set of LLM parameters to be fully loaded into GPU memory before inference can begin. In contrast, PipeBoost allows different GPUs to load distinct portions of the LLM layers, enabling inference to start without waiting for the full model to be loaded.
• Figure 3: PipeBoost overview. Red rectangles and arrows are new designs introduced by PipeBoost. Model checkpoints or parameters are stored on CPU DRAM, local NVMe SSDs or remote storage, e.g., NVMe over Fabrics (NoF), and are loaded into GPU during LLM inference cold starts.
• Figure 4: Different methods of LLM model inference during LLM cold starts on multiple GPUs.
• Figure 5: Details of epoch-based adapter switching.