Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use

TL;DR

This study reveals that LLM inference energy efficiency hinges as much on system design as on model internals. By dissecting inference into prefill and decode phases, the authors show precision benefits primarily in compute-bound prefill, while decode remains memory-bound and less responsive to quantization. They demonstrate that batching and, especially, arrival shaping with Hugging Face's Text Generation Inference can yield dramatic energy reductions—up to two orders of magnitude in per-request energy and substantial daily savings at scale. The findings advocate for phase-aware energy profiling and holistic, stack-wide optimizations to enable sustainable real-world LLM deployment without altering model architectures. The work provides practical guidelines and open-source tooling to measure and optimize energy across the serving stack.

Abstract

Large Language Models (LLMs) are increasingly deployed in production, contributing towards shifting the burden in terms of computational resources and energy demands from training to inference. While prior work has examined the energy cost of inference per prompt or per token, we highlight how \emph{system-level design choices} - such as numerical precision, batching strategy, and request scheduling - can lead to orders-of-magnitude differences in energy consumption for the same model. We perform a detailed empirical study of LLM inference energy and latency on NVIDIA H100 GPUs, analyzing the impact of quantization, batch size, and serving configuration (e.g., with Hugging Face's Text Generation Inference server). Our results reveal that lower-precision formats only yield energy gains in compute-bound regimes; that batching improves energy efficiency, especially in memory-bound phases like decoding; and that structured request timing (arrival shaping) can reduce per-request energy by up to 100 times. We argue that sustainable LLM deployment depends not only on model internals, but also on the orchestration of the serving stack. Our findings motivate phase-aware energy profiling and system-level optimizations for greener AI services.

Figures (7)

• Figure 1: Impact of model size and numerical precision (dtype) on GPU energy consumption during (a) prefill and (b) decode phases.
• Figure 2: GPU energy consumption per token on LLaMA 3.1–8B. (a) Input-side energy depends on token type and padding; (b) Output-side energy remains consistent across requests.
• Figure 3: Impact of inter-arrival delay and model size on energy per request. (a) and (b): Mean energy for LLaMA 8B and 70B under random delays. (c): Comparison of fixed vs. random delays at 8B scale.
• Figure 4: Mean latency per request (with variance across runs) for different models and data types during the prefill phase. Lower-precision formats generally reduce latency, with diminishing returns for already small models.
• Figure 5: Mean latency per generated token (with variance across runs) for different models and data types during the decode phase. Memory-bound regimes lead to latency plateaus despite lower precision.