Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models

TL;DR

This study tackles the under-explored issue of inference energy in large language models by conducting a comprehensive benchmarking across decoder-only and encoder-decoder LLMs, multiple NLP tasks, prompts, and system configurations. It reveals that energy consumption correlates strongly with response time and output length, while input length and task complexity have comparatively smaller effects under fixed input/output lengths. The work demonstrates practical energy-reduction strategies, including quantization, larger but memory-constrained batches, and concise prompts, and introduces a Normalized Accuracy metric to compare cross-task performance. The findings offer actionable guidance for energy-efficient deployment in both local and, to some extent, API-based settings, while acknowledging limitations and ethical considerations of large-scale energy benchmarking.

Abstract

Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.

Figures (9)

• Figure 1: Inference energy vs response time, input and output-token length averaged across samples in a batch plotted across all datasets for Mistral-7B. Dots correspond to distinct batches of different datasets.
• Figure 2: Inference energy on CNN-DM where we vary input token lengths fixing #output tokens to 1.
• Figure 3: Inference energy on CNN-DM dataset when the output length is varied, keeping input length fixed.
• Figure 4: Average per-prompt inference energy vs model size for all models and datasets. The black lines join the median energy for each model family.
• Figure 5: Per-sample inference energy with 4-bit quantized models when the batch size is varied, averaged across all datasets.