Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, Jeff Dean

TL;DR

If the whole ML field adopts best practices, it is predicted that by 2030, total carbon emissions from training will decline.

Abstract

Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use (across research, development, and production) held steady at <15% of Google's total energy use for the past three years. If the whole ML field were to adopt best practices, total carbon emissions from training would reduce. Hence, we recommend that ML papers include emissions explicitly to foster competition on more than just model quality. Estimates of emissions in papers that omitted them have been off 100x-100,000x, so publishing emissions has the added benefit of ensuring accurate accounting. Given the importance of climate change, we must get the numbers right to make certain that we work on its biggest challenges.

The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink

TL;DR

If the whole ML field adopts best practices, it is predicted that by 2030, total carbon emissions from training will decline.

Abstract

Machine Learning (ML) workloads have rapidly grown in importance, but raised concerns about their carbon footprint. Four best practices can reduce ML training energy by up to 100x and CO2 emissions up to 1000x. By following best practices, overall ML energy use (across research, development, and production) held steady at <15% of Google's total energy use for the past three years. If the whole ML field were to adopt best practices, total carbon emissions from training would reduce. Hence, we recommend that ML papers include emissions explicitly to foster competition on more than just model quality. Estimates of emissions in papers that omitted them have been off 100x-100,000x, so publishing emissions has the added benefit of ensuring accurate accounting. Given the importance of climate change, we must get the numbers right to make certain that we work on its biggest challenges.
Paper Structure (9 sections, 3 equations, 3 figures)

This paper contains 9 sections, 3 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Overview of Energy and $\mathrm{CO}_{2} \mathrm{e}$ for ML Training
  3. Case Study 1: Transformer vs. Evolved Transformer vs. Primer
  4. Case Study 2: GPT-3 vs GLaM
  5. Overall ML Energy Consumption
  6. Additional Factors
  7. Related Work
  8. Conclusion
  9. References

Figures (3)

  • Figure 1: Reduction in gross CO2 emissions since 2017 from applying best practices (Section 3). They show large end-to-end improvements, broken down into the$\mathbf{4 M s}$. Gross $\mathrm{CO}_{2}$ emissions here excludes Google's carbon neutral and $\mathbf{1 0 0} \boldsymbol{\%}$ renewable energy credits, and reflect Google's 24/7 CFE methodology [5].
  • Figure 2: Percent Carbon Free Energy by Google Cloud Location in 2020. The map shows the %CFE and how the percentage changes by time of day. Chile has a high %CFE from 6AM to 8PM, but not at night. The US examples on this map range from$19 \%$ CFE in Nevada to $93 \%$ in lowa, which has strong prevailing winds both night and day. (sustainability.google/progress/energy/)
  • Figure 3: Parameters, accelerator years of computation, energy consumption, and gross$\mathrm{CO}_{2} \mathrm{e}$ for GPT-3 and GLaM. If instead of outperforming GPT-3 on quality scores, GLaM was only trained to match, it would halve the time, energy, and CO2e. Google's renewable energy purchases further reduce the impact to zero.