TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems

TL;DR

TokenSim addresses the need for dynamic, QoS-aware modeling of LLM inference across heterogeneous hardware and software configurations. It introduces a transformer-oriented, SimPy-based simulator with extensible scheduling and memory management, and supports disaggregated prefill and decoding phases and real-data workload input. The paper validates TokenSim against real hardware, showing errors below $<1\%$ and demonstrates superior accuracy relative to Vidur and LLMServingSim, along with detailed analyses of continuous batching, memory management, disaggregation, and memory caching. The findings yield practical guidance for hardware choices and system-level optimizations in LLM serving, including when to use memory pools, how to set prefill/decode device ratios, and how memory bandwidth and capacity influence performance.

Abstract

The increasing demand for large language model (LLM) serving has necessitated significant advancements in the optimization and profiling of LLM inference systems. As these models become integral to a wide range of applications, the need for efficient and scalable serving solutions has grown exponentially. This work introduces TokenSim, a comprehensive hardware and software exploration system designed specifically for LLM inference. TokenSim is characterized by its support for extensible system optimizations including scheduling and memory management. We validate the results with systems running with realworld datasets, achieving an error rate of less than 1%. Furthermore, TokenSim facilitates various insightful explorations into the performance and optimization of LLM serving systems.

Figures (15)

• Figure 1: TokenSim system overview.
• Figure 2: One example of TokenSim configurations.
• Figure 3: User-defined scheduler example of global scheduler and local scheduler, defining a disaggregated architecture.
• Figure 4: vLLM throughput and latency validation. "V-" and "T-" stands for vLLM and TokenSim respectively. "Thr" denotes throughput.
• Figure 5: vLLM latency CDF aligns with TokenSim at different QPS, dashed lines are vLLM and solid lines are TokenSim.