Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures

TL;DR

The paper addresses the energy efficiency of DL inference across diverse runtime infrastructures, including PyTorch, TensorFlow, MXNet, and ONNX Runtime, and examines how ONNX and execution providers like CUDA and TensorRT influence performance and energy for ResNet-50, MobileNetV2, and BERT-base-uncased. Using two batch sizes and careful energy measurements on an RTX 3070, the study reveals a near-perfect correlation between total energy and inference time ($\rho=0.99$, $p<0.0001$) and demonstrates that no single framework consistently dominates across models and configurations; MXNet can excel for vision at batch size 1, while PyTorch can be best for BERT, and ONNX often—but not always—yields improvements. TensorRT generally outperforms CUDA in both energy and latency, with substantial gains for MXNet and PyTorch and mixed outcomes for TensorFlow. The work highlights the nuanced and model-dependent nature of energy efficiency in DL runtime infrastructures and points to future directions, including broader model coverage, multiple client languages, and deeper exploration of runtime optimizations. Overall, practitioners should empirically evaluate frameworks, ONNX, and execution providers for their specific use cases to balance energy, latency, and accuracy.

Abstract

Deep Learning (DL) frameworks such as PyTorch and TensorFlow include runtime infrastructures responsible for executing trained models on target hardware, managing memory, data transfers, and multi-accelerator execution, if applicable. Additionally, it is a common practice to deploy pre-trained models on environments distinct from their native development settings. This led to the introduction of interchange formats such as ONNX, which includes its runtime infrastructure, and ONNX Runtime, which work as standard formats that can be used across diverse DL frameworks and languages. Even though these runtime infrastructures have a great impact on inference performance, no previous paper has investigated their energy efficiency. In this study, we monitor the energy consumption and inference time in the runtime infrastructures of three well-known DL frameworks as well as ONNX, using three various DL models. To have nuance in our investigation, we also examine the impact of using different execution providers. We find out that the performance and energy efficiency of DL are difficult to predict. One framework, MXNet, outperforms both PyTorch and TensorFlow for the computer vision models using batch size 1, due to efficient GPU usage and thus low CPU usage. However, batch size 64 makes PyTorch and MXNet practically indistinguishable, while TensorFlow is outperformed consistently. For BERT, PyTorch exhibits the best performance. Converting the models to ONNX yields significant performance improvements in the majority of cases. Finally, in our preliminary investigation of execution providers, we observe that TensorRT always outperforms CUDA.

Figures (1)

• Figure 1: Schematic representation of this study.