Splitwiser: Efficient LM inference with constrained resources

TL;DR

Problem: LLM inference suffers from inefficiencies in separating prompt processing and token generation, limiting hardware utilization. Approach: Splitwiser runs the two inference phases in parallel on a single GPU using multiprocessing and NVIDIA MPS, with implementations for HuggingFace and vLLM to overlap work and reduce data-transfer overhead. Contributions: profiling methodology and extensive experiments demonstrate latency and throughput improvements (e.g., up to 1.42x speedups with vLLM+MPS, 1.1x throughput increase for HuggingFace, and ~18% latency reductions in certain configurations), along with an analysis of scheduler limitations and proposed future designs like on-demand prompt processing. Significance: enables more efficient, cost-effective LLM serving on resource-constrained hardware, broadening accessibility of split-inference techniques and guiding future scheduler and memory-management enhancements.

Abstract

Efficient inference of LLMs remains a crucial challenge, with two main phases: a compute-intensive prompt computation and a memory-intensive token generation. Despite existing batching and scheduling techniques, token generation phases fail to fully utilize compute resources, especially when compared to prompt computation phases. To address these challenges, we propose Splitwiser, a methodology that splits the two phases of an LLM inference request onto the same GPU, thereby reducing overhead and improving memory access and cache utilization. By eliminating the need to transfer data across devices, Splitwiser aims to minimize network-related overheads. In this report, we describe the basic structure of our proposed pipeline while sharing preliminary results and analysis. We implement our proposed multiprocessing design on two widely-used and independent LLM architectures: Huggingface and vLLM. We open-source our code for the respective implementations: 1) Huggingface (https://github.com/asad-aali/splitwiser), and 2) vLLM (https://github.com/adney11/vllm-sysml).

Figures (13)

• Figure 1: Splitwiser design for running prompt processing and token generation in parallel, for achieving inference speedup on a single GPU.
• Figure 2: SM throughput comparison with an increasing number of input tokens demonstrates compute intensity in the prompt processing phase.
• Figure 3: Memory throughput doesn't change, but increasing output tokens results in sustained memory usage.
• Figure 4: SM and memory throughputs running a batch of 5 inference requests. Throughputs are similar to single inference and are not fully utilized.
• Figure 5: KV-Cache usage (%) for a range of batch sizes, for the prompt and token phase.