Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

TL;DR

Etalon addresses the gap between existing latency/throughput metrics and real user experience in LLM inference by introducing fluidity-index, a deadline-based metric that captures how consistently tokens are generated during streaming decoding. Coupled with fluid token generation rate, Etalon provides a black-box, workload-agnostic framework that evaluates end-to-end user perception, capacity, and stalls across open-source and model-as-a-service platforms. The framework is implemented by extending LLMPerf and applying two evaluation modes (black-box and capacity) to both public APIs and open-source systems, revealing nuanced performance trade-offs overlooked by traditional metrics. This work offers a practical standard for user-centric benchmarking and informs deployment decisions and capacity planning in production LLM serving. The availability of Etalon at https://github.com/project-etalon/etalon underpins its adoption as a standard evaluation suite for LLM inference systems.

Abstract

Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon.

Figures (6)

• Figure 1: Increase in prefill latency with prompt length (Yi-34B on 2-H100) makes it infeasible to operate with fixed TTFT SLOs, especially for models with long context support.
• Figure 2: Normalized latency metric could be misleading as it obfuscates scheduling delay. On arxiv_summarization trace, 1.5 QPS, Yi-34B on 2-H100, while the scheduling delay is above 25s for 60% requests in vLLM, the normalized latency only differs by few hundred ms.
• Figure 3: (a) Decode tokens can be intermittently stalled due to prefills from incoming requests. (b) Naively normalizing total decode latency in TPOT, hides these latency spikes and overestimates the system token throughput. (c) Simply observing tail latency does not capture the nuances in the latency distribution. P85 latency for Sarathi-Serve is higher compared to vLLM while it has much lower P99 latency. Performance evaluations with fluid token generation rate accounts for all these variations and provides an accurate and balanced view of system performance. Here for fluid token generation rate, we enforce that 99% of the requests meet deadlines at least 90% of the time (fluidity-index > 0.9).
• Figure 4: When a request arrives in the system, Etalon sets the deadlines for all future tokens. If a token is produced before the set deadline, the slack is carried forward and serves as a buffer for future tokens. When a token arrives late, the system gets penalized for all the missed deadlines, and the subsequent deadlines are reset to account for autoregressive decoding process.
• Figure 5: Evaluation of proprietary serving offerings for Mixtral-8x7B and Llama3-70B performed over duration of 24 hrs. (a) shows the token throughput as estimated by different decode latency metrics, (b) presents the overall decode latency distribution across all requests, (c) shows the TTFT for different prompt lengths and (d) provides a full characterization of the system by showing the fluidity-index as a function of target TBT.