Evaluating the Energy Consumption of Machine Learning: Systematic Literature Review and Experiments

TL;DR

This work addresses the lack of a universal method for evaluating ML energy consumption by performing a broad systematic literature review and an experimental comparison across diverse ML tasks. It introduces a formal protocol to collect, classify, and extract data from energy-evaluation studies, and provides open-source repositories to reproduce and extend the analysis. The study catalogs four main approaches—measurement, data-based estimation, analytical estimation, and on-chip sensors—and shows how they differ in context, inputs, and hardware coverage. Through experiments on MNIST, CIFAR-10, ImageNet, and SQUAD, the authors compare multiple tools against a ground-truth epm baseline, highlighting where certain methods align well with real measurements and where inaccuracies persist. The work highlights the need for standardized, accessible energy-evaluation tools and data to guide ML practitioners toward energy-efficient models and deployments, with a clear path for extending the review as hardware and tasks evolve.

Abstract

Monitoring, understanding, and optimizing the energy consumption of Machine Learning (ML) are various reasons why it is necessary to evaluate the energy usage of ML. However, there exists no universal tool that can answer this question for all use cases, and there may even be disagreement on how to evaluate energy consumption for a specific use case. Tools and methods are based on different approaches, each with their own advantages and drawbacks, and they need to be mapped out and explained in order to select the most suitable one for a given situation. We address this challenge through two approaches. First, we conduct a systematic literature review of all tools and methods that permit to evaluate the energy consumption of ML (both at training and at inference), irrespective of whether they were originally designed for machine learning or general software. Second, we develop and use an experimental protocol to compare a selection of these tools and methods. The comparison is both qualitative and quantitative on a range of ML tasks of different nature (vision, language) and computational complexity. The systematic literature review serves as a comprehensive guide for understanding the array of tools and methods used in evaluating energy consumption of ML, for various use cases going from basic energy monitoring to consumption optimization. Two open-source repositories are provided for further exploration. The first one contains tools that can be used to replicate this work or extend the current review. The second repository houses the experimental protocol, allowing users to augment the protocol with new ML computing tasks and additional energy evaluation tools.

Figures (11)

• Figure 1: Intersection between our work and the surveys selected by our protocol: Jay et al. '23 jay2023, Pijnacker et al. '23 pijnacker2023, Ergasheva et al. '20 ergasheva2020, Fahad et al. '19 fahad2019, Rieger et al. '17 rieger2017, Mobius et al. '14 mobius2014, Noureddine et al. '13 noureddine2013, Bannour et al. '21 bannour2021, García-martín et al. '19 garcia-martin2019a.
• Figure 2: Caption: execution of the protocol -- part I
• Figure 3: Caption: execution of the protocol -- part II
• Figure 4: Caption: Timeline of the search steps
• Figure 5: Taxonomy of energy evaluation techniques for computing tasks.