Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models

TL;DR

This work introduces a coarse-to-fine benchmarking framework to evaluate inference efficiency of large language model code libraries under offline batch and online serving scenarios. It combines a coarse-grained evaluation across four length-distribution datasets with a fine-grained theoretical and empirical analysis of each Transformer module, introducing arithmetic intensity as a key metric to distinguish compute- vs memory-bound bottlenecks. The study shows that KV-cache strategies and batching (e.g., vLLM, DeepSpeed-MII) yield superior batch and serving performance, while attention and memory-bound components like RoPE and RMSNorm remain critical bottlenecks, especially in decoding. By offering a per-module complexity formulation, empirical time breakdowns, and open-source benchmarks, the paper provides practical guidance for optimizing current libraries and steering future inference algorithm development.

Abstract

In real world, large language models (LLMs) can serve as the assistant to help users accomplish their jobs, and also support the development of advanced applications. For the wide application of LLMs, the inference efficiency is an essential concern, which has been widely studied in existing work, and numerous optimization algorithms and code libraries have been proposed to improve it. Nonetheless, users still find it challenging to compare the effectiveness of all the above methods and understand the underlying mechanisms. In this work, we perform a detailed coarse-to-fine analysis of the inference performance of various code libraries. To evaluate the overall effectiveness, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our experiments yield comprehensive results that are invaluable for researchers to evaluate code libraries and improve inference strategies.

Figures (6)

• Figure 1: The throughput and latency for serving inference of vLLM and MII using LLaMA-2 (7B) under different request frequencies on the Long2Short dataset.
• Figure 2: The distributions of total time and module time for both Transformers and vLLM libraries across different input lengths ranging from 32 to 2048 tokens.
• Figure 3: The distribution of input length and output length of three datasets.
• Figure 4: The throughput and latency for serving inference of vLLM and MII using LLaMA-2 (7B) under different request frequencies on the Short2Short dataset.
• Figure 5: The throughput and latency for serving inference of vLLM and MII using LLaMA-2 (7B) under different request frequencies on the Short2Long dataset.