Optimising for Energy Efficiency and Performance in Machine Learning

TL;DR

This work tackles the rising energy footprint of ML by arguing that inference energy, not just training cost, must be measured and optimised. It introduces ECOpt, a Python framework that uses multi-objective Bayesian optimisation to simultaneously maximise model performance and energy efficiency, producing an interpretable Pareto frontier over hyperparameters. Through extensive experiments across hardware, Transformer text-generation, and CIFAR-10 CNN NAS, the authors show that parameter counts and FLOPs are unreliable proxies for energy, that Transformer energy efficiency is relatively hardware-stable, and that energy–accuracy trade-offs can yield Pareto-optimal architectures (seven CIFAR-10 models) and substantial practical gains (e.g., batch-size optimisation achieving up to 38× energy efficiency improvements). The study argues for publishing energy metrics, demonstrates potential environmental benefits, and provides open-source tooling to foster greener ML practices and regulatory compliance (e.g., EU AI Act).

Abstract

The ubiquity of machine learning (ML) and the demand for ever-larger models bring an increase in energy consumption and environmental impact. However, little is known about the energy scaling laws in ML, and existing research focuses on training cost -- ignoring the larger cost of inference. Furthermore, tools for measuring the energy consumption of ML do not provide actionable feedback. To address these gaps, we developed Energy Consumption Optimiser (ECOpt): a hyperparameter tuner that optimises for energy efficiency and model performance. ECOpt quantifies the trade-off between these metrics as an interpretable Pareto frontier. This enables ML practitioners to make informed decisions about energy cost and environmental impact, while maximising the benefit of their models and complying with new regulations. Using ECOpt, we show that parameter and floating-point operation counts can be unreliable proxies for energy consumption, and observe that the energy efficiency of Transformer models for text generation is relatively consistent across hardware. These findings motivate measuring and publishing the energy metrics of ML models. We further show that ECOpt can have a net positive environmental impact and use it to uncover seven models for CIFAR-10 that improve upon the state of the art, when considering accuracy and energy efficiency together.

Figures (9)

• Figure 1: An overview of the ECOpt system. The ECOpt meter can be used independently of the optimiser, to measure energy efficiency and performance metrics. The ECOpt optimiser performs multi-objective optimisation of the model hyperparameters.
• Figure 2: The cost metrics of the neural network model inference on 10,000 inputs in batches of 100, measured per number of layers on the server. We note that all the cost metrics grow linearly with the number of layers.
• Figure 3: The cost metrics of the CNN model inference on 10,000 inputs in batches of 1,000, measured per number of layers on the server. When scaling the stride, we observe that the number of parameters remains constant while the other cost metrics reduce.
• Figure 4: The throughput of each of the models in Table \ref{['tab:consistency-models']} on the machines listed in Table \ref{['tab:machines']}. Apart from the server outperforming the HPC for smaller models, the results are as expected: more powerful hardware produces more tokens per second. The desktop GPU cannot fit Gemma 3 or Llama 3.1 and is thus excluded from the results of these models.
• Figure 5: The energy efficiency of each of the models in Table \ref{['tab:consistency-models']} on the machines listed in Table \ref{['tab:machines']}. While there is a marked difference in energy efficiency between the models, they each show relatively consistent efficiency across machines. The CPU-based machines show a clear advantage for the sub-billion-parameter models, but perform worse with the larger models. Again, the desktop is excluded from the Gemma 3 and Llama 3.1 results due to limited VRAM.