Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Small Talk, Big Impact: The Energy Cost of Thanking AI

Julien Delavande, Regis Pierrard, Sasha Luccioni

TL;DR

This work quantifies the energy cost of polite interactions with large language models by measuring energy use during thousands of polite completions on open-source LLMs. It introduces a two-phase decomposition (prefill and decode) and a closed-form latency model that links input/output lengths to energy through compute- and memory-bound regimes, validated by empirical data from an H100-based setup. Key findings show energy grows linearly with prompt and generation lengths, larger models incur substantially higher energy due to deeper architectures and longer outputs, and prefill dominates upfront while decode drives the tail; these insights enable more sustainable deployment strategies. The study provides reproducible measurement tooling and latency models, highlighting a practical trade-off between politeness (which can improve alignment and user trust) and energy efficiency in real-world AI systems.

Abstract

Being polite is free - or is it? In this paper, we quantify the energy cost of seemingly innocuous messages such as ``thank you'' when interacting with large language models, often used by users to convey politeness. Using real-world conversation traces and fine-grained energy measurements, we quantify how input length, output length and model size affect energy use. While politeness is our motivating example, it also serves as a controlled and reproducible proxy for measuring the energy footprint of a typical LLM interaction. Our findings provide actionable insights for building more sustainable and efficient LLM applications, especially in increasingly widespread real-world contexts like chat. As user adoption grows and billions of prompts are processed daily, understanding and mitigating this cost becomes crucial - not just for efficiency, but for sustainable AI deployment.

Small Talk, Big Impact: The Energy Cost of Thanking AI

TL;DR

This work quantifies the energy cost of polite interactions with large language models by measuring energy use during thousands of polite completions on open-source LLMs. It introduces a two-phase decomposition (prefill and decode) and a closed-form latency model that links input/output lengths to energy through compute- and memory-bound regimes, validated by empirical data from an H100-based setup. Key findings show energy grows linearly with prompt and generation lengths, larger models incur substantially higher energy due to deeper architectures and longer outputs, and prefill dominates upfront while decode drives the tail; these insights enable more sustainable deployment strategies. The study provides reproducible measurement tooling and latency models, highlighting a practical trade-off between politeness (which can improve alignment and user trust) and energy efficiency in real-world AI systems.

Abstract

Being polite is free - or is it? In this paper, we quantify the energy cost of seemingly innocuous messages such as ``thank you'' when interacting with large language models, often used by users to convey politeness. Using real-world conversation traces and fine-grained energy measurements, we quantify how input length, output length and model size affect energy use. While politeness is our motivating example, it also serves as a controlled and reproducible proxy for measuring the energy footprint of a typical LLM interaction. Our findings provide actionable insights for building more sustainable and efficient LLM applications, especially in increasingly widespread real-world contexts like chat. As user adoption grows and billions of prompts are processed daily, understanding and mitigating this cost becomes crucial - not just for efficiency, but for sustainable AI deployment.
Paper Structure (27 sections, 14 equations, 6 figures)

This paper contains 27 sections, 14 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset and Experimental Setup
  4. Hardware and Software Environment
  5. Models and Precision
  6. The Energy Cost of a Simple "Thank You"
  7. Component-wise Energy Consumption
  8. Energy and Latency Dependence on Input and Output Length
  9. Theoretical Latency Model
  10. Prefill phase.
  11. Decode phase.
  12. Link to Empirical Energy Trends
  13. Prefill energy.
  14. Decode energy.
  15. No quadratic growth.
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Distribution of GPU energy consumption across "thank you" generations. The long tail indicates variability due to prompt and output length.
  • Figure 2: Energy consumption by hardware type (GPU, CPU, RAM) and phase (prefill, decode, generate). The GPU consistently dominates across all phases, while CPU and RAM consumption remains minor.
  • Figure 3: Histogram of GPU energy consumption for prefill and decode phases. Decode contributes most to the long tail due to repeated sequential steps, whereas prefill energy is substantial but occurs only once per request.
  • Figure 4: Correlation between token lengths and GPU energy consumption in prefill and decode phases.
  • Figure 5: GPU energy consumption during generation as a function of model size. Boxes represent the distribution across 10,000 replies including polite "thank you" phrases.
  • ...and 1 more figures