Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Bin Batching for Increasing LLM Inference Throughput

Ozgur Guldogan, Jackson Kunde, Kangwook Lee, Ramtin Pedarsani

TL;DR

This work tackles inefficiencies in batched LLM inference caused by variable output lengths by introducing Multi-Bin Batching, which partitions requests into $k$ bins based on predicted execution times and batches within each bin. Through a queueing-theoretic analysis, it proves asymptotic throughput optimality as $k$ grows, deriving a closed-form expression for throughput and establishing how many bins are needed to achieve near-maximum capacity; experiments with real models show throughput gains up to about 70% under oracle-length information. The approach robustly improves CPU/GPU utilization by reducing the impact of long-tail generation times, while latency trade-offs are analyzed under an infinite-server approximation. The results suggest a practical, deployable scheduling policy that can be integrated into LLM serving stacks to substantially boost inference throughput, with potential extensions to adaptive binning and improved length prediction.

Abstract

As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches.

Multi-Bin Batching for Increasing LLM Inference Throughput

TL;DR

This work tackles inefficiencies in batched LLM inference caused by variable output lengths by introducing Multi-Bin Batching, which partitions requests into bins based on predicted execution times and batches within each bin. Through a queueing-theoretic analysis, it proves asymptotic throughput optimality as grows, deriving a closed-form expression for throughput and establishing how many bins are needed to achieve near-maximum capacity; experiments with real models show throughput gains up to about 70% under oracle-length information. The approach robustly improves CPU/GPU utilization by reducing the impact of long-tail generation times, while latency trade-offs are analyzed under an infinite-server approximation. The results suggest a practical, deployable scheduling policy that can be integrated into LLM serving stacks to substantially boost inference throughput, with potential extensions to adaptive binning and improved length prediction.

Abstract

As large language models (LLMs) grow in popularity for their diverse capabilities, improving the efficiency of their inference systems has become increasingly critical. Batching LLM requests is a critical step in scheduling the inference jobs on servers (e.g. GPUs), enabling the system to maximize throughput by allowing multiple requests to be processed in parallel. However, requests often have varying generation lengths, causing resource underutilization, as hardware must wait for the longest-running request in the batch to complete before moving to the next batch. We formalize this problem from a queueing-theoretic perspective, and aim to design a control policy which is throughput-optimal. We propose Multi-Bin Batching, a simple yet effective method that can provably improve LLM inference throughput by grouping requests with similar (predicted) execution times into predetermined bins. Through a combination of theoretical analysis and experiments, including real-world LLM inference scenarios, we demonstrate significant throughput gains compared to standard batching approaches.

Paper Structure

This paper contains 23 sections, 7 theorems, 54 equations, 11 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Inference and Scheduling.
  4. LLM Serving and Answer Length Estimation.
  5. LLM Inference Optimization.
  6. Continuous Batching.
  7. Problem Setup and the Multi-Bin Batching Algorithm
  8. Multi-Bin Batching Algorithm
  9. Throughput Analysis
  10. Latency Analysis
  11. LLM Experiments
  12. Simulated LLM Inference
  13. Effect of Symmetrical Prediction Errors
  14. Conclusion
  15. Proofs for Throughput Analysis
  16. ...and 8 more sections

Key Result

Proposition 4.1

The expected throughput of the system is the ratio of the batch size to the expected service time. Specifically, it can be written as where $B$ is the batch size and $\mathbb{E}[t_{\text{service}}]$ is the expected service time of a batch of $B$ requests.

Figures (11)

  • Figure 1: \ref{['fig:batching_throughput']} Batch serving improves the throughput for the LLM inference systems. \ref{['fig:batch_inefficiency']} Standard batching causes underutilization of resources due to varying answer sizes.
  • Figure 2: k-Bin Batching
  • Figure 3: Average throughput of $k$-bin batching vs the arrival rate.
  • Figure 4: The latency of a request vs the arrival rate $\lambda$ for different $k$ values.
  • Figure 5: The linear relationship between the number of tokens generated and the inference time
  • ...and 6 more figures

Theorems & Definitions (16)

  • Proposition 4.1
  • Remark 4.1
  • Lemma 4.1
  • Theorem 4.2
  • Remark 4.2
  • Remark 4.3
  • Theorem 4.3
  • Lemma 5.1
  • proof
  • proof
  • ...and 6 more