The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization

TL;DR

The ML.ENERGY Benchmark tackles the energy bottleneck in real-world generative AI by providing an extensible benchmark and leaderboard that measure inference energy in deployment-like settings. It combines two core capabilities: service-aware energy accounting across diffusion and LLM tasks, and automated optimization that derives energy-saving configurations while respecting latency targets. The design emphasizes generalizability, realistic deployments, appropriate measurement granularity, and actionable results, demonstrated by early-2025 results across 40 models and 6 tasks with notable energy reductions. By enabling cross-layer insights and open-source tooling, the work aims to democratize energy-aware deployment of generative AI and inform hardware/software co-design for greener AI services.

Abstract

As the adoption of Generative AI in real-world services grow explosively, energy has emerged as a critical bottleneck resource. However, energy remains a metric that is often overlooked, under-explored, or poorly understood in the context of building ML systems. We present the ML$.$ENERGY Benchmark, a benchmark suite and tool for measuring inference energy consumption under realistic service environments, and the corresponding ML$.$ENERGY Leaderboard, which have served as a valuable resource for those hoping to understand and optimize the energy consumption of their generative AI services. In this paper, we explain four key design principles for benchmarking ML energy we have acquired over time, and then describe how they are implemented in the ML$.$ENERGY Benchmark. We then highlight results from the early 2025 iteration of the benchmark, including energy measurements of 40 widely used model architectures across 6 different tasks, case studies of how ML design choices impact energy consumption, and how automated optimization recommendations can lead to significant (sometimes more than 40%) energy savings without changing what is being computed by the model. The ML$.$ENERGY Benchmark is open-source and can be easily extended to various customized models and application scenarios.

Figures (12)

• Figure 1: Overview of the benchmarking and optimization flow of the ML.ENERGY Benchmark.
• Figure 2: LLM inference server and per-request energy accounting. The steady state is defined as the period when batch size is saturated at the server's maximum configured batch size, and measurements during the steady state represent that of a serving system during long-term deployment.
• Figure 3: Per-request energy consumption across various generative AI models. Black and orange represents text and vision modalities, respectively. Solid bars are energy measurements, whereas dimmed bars behind each solid bar are estimations based on the GPU's TDP, with numbers showing the ratio of overestimation. Note the log scale Y-axis.
• Figure 4: Phi-3 Mini and Small phi3-arxiv24 benchmarked with the chat task on one NVIDIA A100 GPU.
• Figure 5: Power consumption of Llama 3.1 70B and Stable Diffusion 3 Medium models.