Watt For What: Rethinking Deep Learning's Energy-Performance Relationship

TL;DR

A comprehensive study on the electricity consumption of various deep learning models across different GPUs is conducted, presenting a detailed analysis of their accuracy-efficiency trade-offs and highlighting the potential for a more sustainable approach to deep learning.

Abstract

Deep learning models have revolutionized various fields, from image recognition to natural language processing, by achieving unprecedented levels of accuracy. However, their increasing energy consumption has raised concerns about their environmental impact, disadvantaging smaller entities in research and exacerbating global energy consumption. In this paper, we explore the trade-off between model accuracy and electricity consumption, proposing a metric that penalizes large consumption of electricity. We conduct a comprehensive study on the electricity consumption of various deep learning models across different GPUs, presenting a detailed analysis of their accuracy-efficiency trade-offs. By evaluating accuracy per unit of electricity consumed, we demonstrate how smaller, more energy-efficient models can significantly expedite research while mitigating environmental concerns. Our results highlight the potential for a more sustainable approach to deep learning, emphasizing the importance of optimizing models for efficiency. This research also contributes to a more equitable research landscape, where smaller entities can compete effectively with larger counterparts. This advocates for the adoption of efficient deep learning practices to reduce electricity consumption, safeguarding the environment for future generations whilst also helping ensure a fairer competitive landscape.

Figures (9)

• Figure 1: Bridging the Energy Divide: Deep Learning Models vs. Everyday Power Hogs. For easy comparison, we list the amount of electricity consumed per month by an appliance or by the average household in the UK.
• Figure 2: Comparing accuracy and electricity consumption over time. We see that whilst the accuracy of models are improving at a linear rate, the electricity consumed to train them is exponential. The electricity is measured in kWh.
• Figure 3: Using SAM achieves a better balance between accuracy and electricity. In (a), lower accuracy models like MobileNet, ResNet, and EfficientNet are depicted poor compared to ViT-H. In (b), ViT-H and CLIP are penalized for high electricity usage, while EfficientNet and MobileNet rise. Marker points = electricity consumption (Zoom for details).
• Figure 4: Comparing the cost of self-supervised pre-training and then fine-tuning along with inference of BEiT and MAE in terms of electricity consumption. The inference cost is so low in comparison that it is not even visible in the graph.
• Figure 5: Comparing the effect of pre-training on Swin and ViT in terms of accuracy and electricity consumption.