Where Do the Joules Go? Diagnosing Inference Energy Consumption

TL;DR

The paper tackles the growing challenge of inference energy consumption in generative AI by first measuring time and energy at scale across 46 models, 7 tasks, and 1,858 configurations on NVIDIA H100 and B200 GPUs, revealing energy differences up to $25\times$ (and higher across modalities). It then introduces a latent-factor framework that links wall-clock time and energy to underlying factors such as memory, utilization, and system bottlenecks, extending the view to throughput per watt for datacenter sizing. Key findings show that decoding dominates LLM energy, longer outputs stress memory and batch size, CPU-side preprocessing can bottleneck multimodal models, diffusion energy depends strongly on denoising steps and frames, and that precision and multi-GPU scaling yield nontrivial tradeoffs. The framework enables principled optimization across model choice, precision, hardware, and deployment constraints, with practical implications for energy-aware AI datacenters and higher work-per-watt efficiency in real systems.

Abstract

Energy is now a critical ML computing resource. While measuring energy consumption and observing trends is a valuable first step, accurately understanding and diagnosing why those differences occur is crucial for optimization. To that end, we begin by presenting a large-scale measurement study of inference time and energy across the generative AI landscape with 46 models, 7 tasks, and 1,858 different configurations on NVIDIA H100 and B200 GPUs. Our empirical findings span order-of-magnitude variations: LLM task type can lead to 25$\times$ energy differences, video generation sometimes consumes more than 100$\times$ the energy of images, and GPU utilization differences can result in 3--5$\times$ energy differences. Based on our observations, we present a framework for reasoning about the underlying mechanisms that govern time and energy consumption. The essence is that time and energy are determined by latent metrics like memory and utilization, which are in turn affected by various factors across the algorithm, software, and hardware layers. Our framework also extends directly to throughput per watt, a critical metric for power-constrained datacenters.

Figures (12)

• Figure 1: Distribution of (a) number of output tokens, (b) energy per token, and (c) energy per response across all models on B200 GPUs using their respective minimum-energy configurations.
• Figure 2: (a) shows three models from the Qwen 3 family across three modalities (minimum-energy configurations), and (b) shows how modality affects batch size and KV cache utilization for the 8B model, showing why energy per token increases. Text modality uses the text-only model (e.g., Qwen 3 8B), whereas Image and Video use the vision--language variant (e.g., Qwen 3 VL 8B).
• Figure 3: Energy per image/video for diffusion models (minimum-energy configuration on B200). SD is short for Stable Diffusion.
• Figure 4: Energy per token, throughput, median ITL, and power trends against batch size for (a) DeepSeek R1 (Problem Solving) on 8$\times$ B200 and (b) Qwen 3 Coder 30B A3B (Code Completion) on 1$\times$ B200. Metrics normalized to % of maximum, except power which is normalized to % of GPU TDP.
• Figure 5: Energy/token by active parameters of Problem Solving models with the minimum-energy configuration on B200 GPUs.