Reasoning Language Model Inference Serving Unveiled: An Empirical Study

TL;DR

This work investigates the real-world serving behavior of reasoning large language models ($RLLM$) and compares them to traditional LLMs. It introduces the ASU assessment framework and the ASU-Perf benchmark, then conducts pilot studies and extensive optimizations, evaluating model weight quantization, KV-cache quantization, prefix caching, and speculative decoding. Key findings show pronounced memory fluctuations, straggler-induced delays, and adaptive runtimes in $RLLM$ serving, with mixed results for optimization techniques depending on model size; real-world Gamma workloads corroborate the observed patterns. The study provides actionable guidance for deploying efficient $RLLM$ inference and highlights gaps in current serving ecosystems that researchers and industry should address to democratize access to advanced reasoning models.

Abstract

The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving performance and behavior of RLLM remains unexplored, which may undermine the deployment and utilization of RLLM in real-world scenario. To close this gap, in this paper, we conduct a comprehensive study of RLLM service. We first perform a pilot study on comparing the serving performance between RLLM and traditional LLM and reveal that there are several distinct differences regarding serving behavior: (1) significant memory usage and fluctuations; (2) straggler requests; (3) adaptive running time; (4) domain preference. Then we further investigate whether existing inference optimization techniques are valid for RLLM. Our main takeaways are that model quantization methods and speculative decoding can improve service system efficiency with small compromise to RLLM accuracy, while prefix caching, KV cache quantization may even degrade accuracy or serving performance for small RLLM. Lastly, we conduct evaluation under real world workload modeled by Gamma distribution to verify our findings. Empirical results of real world workload evaluation across different dataset are aligned with our main findings regarding RLLM serving. We hope our work can provide the research community and industry with insights to advance RLLM inference serving.

Figures (15)

• Figure 1: Results of token budget variation across different datasets for 14B and 32B RLLM .
• Figure 2: The serving performance and behavior comparison of a batch requests between 7B RLLM and LLM. We can read from this figure that (1) RLLM exhibits significant KV Cache fluctuations than LLM; (2) long tail distribution of requests running time caused by straggler requests; (3) adaptive running time of RLLM; (4) domain preference on math. Please refer to § \ref{['app:serving-behaviors-pilot-study']} for more results.
• Figure 3: Empirical results of current LLM quantization methods on 7B RLLM. current methods maintain or improve all serving-related metrics with less memory footprint while keep accuracy.
• Figure 4: Empirical results for KV cache quantization on 14B model across different datasets.
• Figure 5: Empirical results of comparison for enable or disable prefix caching on 32B RLLM.