Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Price of Prompting: Profiling Energy Use in Large Language Models Inference

Erik Johannes Husom, Arda Goknil, Lwin Khin Shar, Sagar Sen

TL;DR

MELODI introduces a real-time energy-monitoring framework for LLM inference by pairing Scaphandre (CPU power) and Nvidia-smi (GPU power) to quantify energy use across diverse hardware, models, and prompt datasets. The authors create a rich energy dataset and perform analyses linking energy consumption to prompt and response characteristics, revealing that longer responses drive energy use more than prompt complexity and that predictive models based on response features generalize well across configurations. Key findings show substantial energy disparities across model sizes and hardware, with large 70B models consuming ~100x more energy per token than small 7B models, and CPU-only laptops exhibiting less reliable energy estimates. The work provides actionable insights and data to guide energy-conscious LLM deployment, including the potential to reduce energy costs by constraining response length and prioritizing GPU-accelerated configurations.

Abstract

In the rapidly evolving realm of artificial intelligence, deploying large language models (LLMs) poses increasingly pressing computational and environmental challenges. This paper introduces MELODI - Monitoring Energy Levels and Optimization for Data-driven Inference - a multifaceted framework crafted to monitor and analyze the energy consumed during LLM inference processes. MELODI enables detailed observations of power consumption dynamics and facilitates the creation of a comprehensive dataset reflective of energy efficiency across varied deployment scenarios. The dataset, generated using MELODI, encompasses a broad spectrum of LLM deployment frameworks, multiple language models, and extensive prompt datasets, enabling a comparative analysis of energy use. Using the dataset, we investigate how prompt attributes, including length and complexity, correlate with energy expenditure. Our findings indicate substantial disparities in energy efficiency, suggesting ample scope for optimization and adoption of sustainable measures in LLM deployment. Our contribution lies not only in the MELODI framework but also in the novel dataset, a resource that can be expanded by other researchers. Thus, MELODI is a foundational tool and dataset for advancing research into energy-conscious LLM deployment, steering the field toward a more sustainable future.

The Price of Prompting: Profiling Energy Use in Large Language Models Inference

TL;DR

MELODI introduces a real-time energy-monitoring framework for LLM inference by pairing Scaphandre (CPU power) and Nvidia-smi (GPU power) to quantify energy use across diverse hardware, models, and prompt datasets. The authors create a rich energy dataset and perform analyses linking energy consumption to prompt and response characteristics, revealing that longer responses drive energy use more than prompt complexity and that predictive models based on response features generalize well across configurations. Key findings show substantial energy disparities across model sizes and hardware, with large 70B models consuming ~100x more energy per token than small 7B models, and CPU-only laptops exhibiting less reliable energy estimates. The work provides actionable insights and data to guide energy-conscious LLM deployment, including the potential to reduce energy costs by constraining response length and prioritizing GPU-accelerated configurations.

Abstract

In the rapidly evolving realm of artificial intelligence, deploying large language models (LLMs) poses increasingly pressing computational and environmental challenges. This paper introduces MELODI - Monitoring Energy Levels and Optimization for Data-driven Inference - a multifaceted framework crafted to monitor and analyze the energy consumed during LLM inference processes. MELODI enables detailed observations of power consumption dynamics and facilitates the creation of a comprehensive dataset reflective of energy efficiency across varied deployment scenarios. The dataset, generated using MELODI, encompasses a broad spectrum of LLM deployment frameworks, multiple language models, and extensive prompt datasets, enabling a comparative analysis of energy use. Using the dataset, we investigate how prompt attributes, including length and complexity, correlate with energy expenditure. Our findings indicate substantial disparities in energy efficiency, suggesting ample scope for optimization and adoption of sustainable measures in LLM deployment. Our contribution lies not only in the MELODI framework but also in the novel dataset, a resource that can be expanded by other researchers. Thus, MELODI is a foundational tool and dataset for advancing research into energy-conscious LLM deployment, steering the field toward a more sustainable future.
Paper Structure (19 sections, 4 figures, 5 tables)

This paper contains 19 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models (LLMs)
  4. Monitoring Tools and Technologies
  5. Dataset Collection: MELODI Framework for Monitoring Energy Usage
  6. Dataset Composition
  7. Dataset Analysis
  8. Analysis Setup
  9. Results
  10. RQ1 (energy consumption of LLM inference across different setups)
  11. RQ1 Conclusion.
  12. RQ2 (the relationship between prompt complexity, response characteristics and the energy consumption)
  13. RQ2 Conclusion.
  14. RQ3 (predictive model for the energy consumption of LLMs)
  15. RQ3 Conclusion.
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of the MELODI framework.
  • Figure 2: Comparative energy consumption per token across inference setups.
  • Figure 3: $R^2$ scores of RF models using only response length for forecasting energy consumption per response.
  • Figure 4: R$^2$ scores of RF models using prompt characteristics for forecasting energy consumption per response.