Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training

TL;DR

The paper investigates the environmental footprint of training large language models by quantifying CO2 emissions with CodeCarbon across BERT, DistilBERT, and T5 using different tokenizers on T4 and A100 GPUs. It leverages SQuAD and AdversarialQA benchmarks to assess performance via cosine similarity and STS while tracking emissions and costs, highlighting trade-offs between speed, cost, and environmental impact. Key findings show substantial reductions in training time and CO2 emissions when using the A100 GPU, and that tokenizer and model compression (e.g., DistilBERT) can lower emissions without sacrificing performance. The study proposes practical mitigation strategies, including lightweight models and faster hardware, while discussing affordability and accessibility, to enable sustainable AI training without sacrificing robustness.

Abstract

Prominent works in the field of Natural Language Processing have long attempted to create new innovative models by improving upon previous model training approaches, altering model architecture, and developing more in-depth datasets to better their performance. However, with the quickly advancing field of NLP comes increased greenhouse gas emissions, posing concerns over the environmental damage caused by training LLMs. Gaining a comprehensive understanding of the various costs, particularly those pertaining to environmental aspects, that are associated with artificial intelligence serves as the foundational basis for ensuring safe AI models. Currently, investigations into the CO2 emissions of AI models remain an emerging area of research, and as such, in this paper, we evaluate the CO2 emissions of well-known large language models, which have an especially high carbon footprint due to their significant amount of model parameters. We argue for the training of LLMs in a way that is responsible and sustainable by suggesting measures for reducing carbon emissions. Furthermore, we discuss how the choice of hardware affects CO2 emissions by contrasting the CO2 emissions during model training for two widely used GPUs. Based on our results, we present the benefits and drawbacks of our proposed solutions and make the argument for the possibility of training more environmentally safe AI models without sacrificing their robustness and performance.

Figures (1)

• Figure 1: Flowchart of the process used for training, testing, and recording measurements of model performance.