Beyond Efficiency: Scaling AI Sustainably

TL;DR

The carbon impact of AI is characterized, including both operational carbon emissions from training and inference and embodied carbon emissions from data center construction and hardware manufacturing, to highlight key efficiency optimization opportunities for cutting-edge AI technologies.

Abstract

Barroso's seminal contributions in energy-proportional warehouse-scale computing launched an era where modern datacenters have become more energy efficient and cost effective than ever before. At the same time, modern AI applications have driven ever-increasing demands in computing, highlighting the importance of optimizing efficiency across the entire deep learning model development cycle. This paper characterizes the carbon impact of AI, including both operational carbon emissions from training and inference as well as embodied carbon emissions from datacenter construction and hardware manufacturing. We highlight key efficiency optimization opportunities for cutting-edge AI technologies, from deep learning recommendation models to multi-modal generative AI tasks. To scale AI sustainably, we must also go beyond efficiency and optimize across the life cycle of computing infrastructures, from hardware manufacturing to datacenter operations and end-of-life processing for the hardware.

Figures (6)

• Figure 1: Significant GPU performance growth from higher transistor density, higher frequency, and larger die size.
• Figure 2: (a) Operational carbon impact of open-source AI model training: GPT-3 brown2020language, DLRM and Universal LM wu2022sustainable, GLaM, BLOOM-175B du2022glam, NLLB nllb, Llama-65/33/13/7B touvron2023llama. Open-source models from Meta are marked in green. The carbon impact result assumes fixed carbon intensity for electricity. When considering location-based carbon intensity of electricity, the carbon impact of BLOOM-175B is 30 tonnes CO2eq. In addition, corporate-level sustainability programs mitigate model training's carbon emissions. (b) Large model training resource consumption (normalized to Large Language Model (LLM)): Deep learning recommender systems require more than three times higher training resources than LLM training.
• Figure 3: (a) Lifecycle carbon impact of the Multi-Lingual Language Translation model (Universal LM in Figure \ref{['fig:carbonimpact-ai-ops-all-v2']}) and the Deep Learning Recommendation Model (DLRM-1). Over the model development lifecycle, the operational carbon impact of model training vs. inference is about 1 to 2 for Universal LM and 1 to 1 for DLRM-1. When considering the overall AI system lifecycle, embodied carbon footprint from hardware manufacturing introduces an additional 50% of the operational carbon emissions. Reducing any part of the lifecycle emissions -- Training vs. Inference; Operational vs. Embodied Carbon Emissions -- will translate to lower lifecycle emissions for AI. (b) The power capacity breakdown for the three key phases of the AI development lifecycle -- Experimentation, Training, and Inference.
• Figure 5: (a) Computing's carbon emissions for datacenter computing using the Greenhouse Gas (GHG) Protocol. (b) Computing's carbon emissions for datacenter hardware using Life Cycle Assessment (LCA).
• Figure 6: Average and marginal emissions comparison at CISO & PJM hourly average for year 2022. Error bars represent standard deviation.