How Green are Neural Language Models? Analyzing Energy Consumption in Text Summarization Fine-tuning

TL;DR

The paper investigates the environmental footprint of fine-tuning three language-model architectures (T5-base, BART-base, LLaMA-3-8B) for generating research highlights from abstracts, balancing performance against energy use. It uses the MixSub dataset and a suite of lexical and semantic metrics (ROUGE, METEOR, MoverScore, BERTScore, SciBERTScore) to evaluate outputs, while applying a Green AI-based carbon-footprint estimation to quantify energy costs per epoch. Results show smaller models (T5-base, BART-base) outperform the large LLaMA-3-8B on lexical metrics but LLaMA-3-8B matches semantic similarity; energy costs scale dramatically with model size, with LLaMA-3-8B incurring ~43.98 gCO2e/epoch versus ~3.5 gCO2e and ~2.4 gCO2e for the smaller models. The study advocates energy-efficient NLP practices and highlights the importance of considering environmental impact in model selection and fine-tuning strategies for summarization tasks.

Abstract

Artificial intelligence systems significantly impact the environment, particularly in natural language processing (NLP) tasks. These tasks often require extensive computational resources to train deep neural networks, including large-scale language models containing billions of parameters. This study analyzes the trade-offs between energy consumption and performance across three neural language models: two pre-trained models (T5-base and BART-base), and one large language model (LLaMA-3-8B). These models were fine-tuned for the text summarization task, focusing on generating research paper highlights that encapsulate the core themes of each paper. The carbon footprint associated with fine-tuning each model was measured, offering a comprehensive assessment of their environmental impact. It is observed that LLaMA-3-8B produces the largest carbon footprint among the three models. A wide range of evaluation metrics, including ROUGE, METEOR, MoverScore, BERTScore, and SciBERTScore, were employed to assess the performance of the models on the given task. This research underscores the importance of incorporating environmental considerations into the design and implementation of neural language models and calls for the advancement of energy-efficient AI methodologies.

Figures (4)

• Figure 1: An (abstract, highlights) pair from the MixSub dataset. https://www.sciencedirect.com/science/article/pii/S0001457519307213. We have used colors to denote the correspondence between a highlight and the abstract. Here, we find each highlight to be a segment of some sentence in the abstract. However, not in all papers, is there a straightforward mapping from abstracts to sentences in the abstract. In particular, a highlight may combine information from multiple sentences in the abstract and express it using very different words.
• Figure 2: Highlights generated by the different fine-tuned models are shown. The input is an abstract from the MixSub test split, sourced from https://www.sciencedirect.com/science/article/pii/S0303264720300836.
• Figure 3: Energy consumption by resource (GPU, CPU and memory), and carbon footprints for model fine-tuning for one epoch. The resource-wise shares of energy consumption (leftmost figure) are identical for all our models (GPU: 67.4%, CPU: 24.3%, memory: 8.3%). The three figures on the right show carbon footprints (in gCO$_{2}$e) on the Y-axis for different locations of the data centers on the X-axis, and highlight the location we used. The carbon footprints are 3.5 gCO$_{2}$e for T5-base, 2.4 gCO$_{2}$e for BART-base, and 43.98 gCO$_{2}$e for LLaMA-3-8B.
• Figure 4: Comparison of computational resources utilized during fine-tuning of the PLMs (T5-base and BART-base) and LLaMA-3-8B LLM for the text summarization task.