Determining Energy Efficiency Sweet Spots in Production LLM Inference

TL;DR

This work tackles the non-linear energy footprint of production LLM inference by deriving two analytical models—FLOPs-based and Memory+FLOPs-based—that connect energy per token to Transformer compute and memory traffic. By validating on TensorRT-LLM with NVIDIA H100 across 1B–9B models and inputs/outputs from 64 to 4096 tokens, the authors achieve a mean MAPE of $1.79\%$, demonstrating that energy efficiency follows distinct Sweet Spots rather than a monotonic trend. They derive the optimal output length $n_{\text{out}}^{*} = \sqrt{(\theta_{1} n_{\text{in}}^{2} + \theta_{3} n_{\text{in}} + \theta_{5})/\theta_{4}}$, showing how input costs amortize against decoding costs to minimize energy per token. The findings offer practical guidance for prompt design, truncation, and adaptive generation in real deployments, enabling energy-aware LLM serving and more sustainable AI systems.

Abstract

Large Language Models (LLMs) inference is central in modern AI applications, making it critical to understand their energy footprint. Existing approaches typically estimate energy consumption through simple linear functions of input and output sequence lengths, yet our observations reveal clear Energy Efficiency regimes: peak efficiency occurs with short-to-moderate inputs and medium-length outputs, while efficiency drops sharply for long inputs or very short outputs, indicating a non-linear dependency. In this work, we propose an analytical model derived from the computational and memory-access complexity of the Transformer architecture, capable of accurately characterizing the efficiency curve as a function of input and output lengths. To assess its accuracy, we evaluate energy consumption using TensorRT-LLM on NVIDIA H100 GPUs across a diverse set of LLMs ranging from 1B to 9B parameters, including OPT, LLaMA, Gemma, Falcon, Qwen2, and Granite, tested over input and output lengths from 64 to 4096 tokens, achieving a mean MAPE of 1.79%. Our results show that aligning sequence lengths with these efficiency "Sweet Spots" can substantially reduce energy usage, supporting informed truncation, summarization, and adaptive generation strategies in production systems.

Figures (4)

• Figure 1: Energy Efficiency $\frac{\text{Tokens}}{\text{Joule}}$ of Falcon 7B with different number of requests (10,100,1000).
• Figure 2: Energy Efficiency of tested LLM considering $n_{\text{in}}=64$ and varying output tokens, per size family (XS, S, M).
• Figure 3: Aggregated Energy Efficiency Heatmap.
• Figure 4: Energy efficiency of Llama 3.2 1B varying maximum batch size considering different number of requests.