Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

Zhenxiao Fu, Fan Chen, Shan Zhou, Haitong Li, Lei Jiang

TL;DR

This paper tackles the challenge of accurately estimating the carbon footprint of LLM inferences before execution. It introduces LLMCO2, a graph neural network model that represents transformer layers as kernel graphs and uses Roofline-based hardware features, separating prefill and decode phases to capture distinct energy profiles. A focused data sampling strategy ensures training data reflects real-world configurations, enabling robust predictions across diverse LLMs and GPUs. Empirical results show LLMCO2 substantially outperforms baselines in both MAPE and EBA, highlighting its potential to inform low-carbon deployment and cloud-billing decisions. The approach advances carbon-aware inference by combining kernel-level graphs, hardware-aware features, and targeted data sampling to predict operational energy and CO2eq accurately.

Abstract

Throughout its lifecycle, a large language model (LLM) generates a substantially larger carbon footprint during inference than training. LLM inference requests vary in batch size, prompt length, and token generation number, while cloud providers employ different GPU types and quantities to meet diverse service-level objectives for accuracy and latency. It is crucial for both users and cloud providers to have a tool that quickly and accurately estimates the carbon impact of LLM inferences based on a combination of inference request and hardware configurations before execution. Estimating the carbon footprint of LLM inferences is more complex than training due to lower and highly variable model FLOPS utilization, rendering previous equation-based models inaccurate. Additionally, existing machine learning (ML) prediction methods either lack accuracy or demand extensive training data, as they inadequately handle the distinct prefill and decode phases, overlook hardware-specific features, and inefficiently sample uncommon inference configurations. We introduce \coo, a graph neural network (GNN)-based model that greatly improves the accuracy of LLM inference carbon footprint predictions compared to previous methods.

LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

TL;DR

This paper tackles the challenge of accurately estimating the carbon footprint of LLM inferences before execution. It introduces LLMCO2, a graph neural network model that represents transformer layers as kernel graphs and uses Roofline-based hardware features, separating prefill and decode phases to capture distinct energy profiles. A focused data sampling strategy ensures training data reflects real-world configurations, enabling robust predictions across diverse LLMs and GPUs. Empirical results show LLMCO2 substantially outperforms baselines in both MAPE and EBA, highlighting its potential to inform low-carbon deployment and cloud-billing decisions. The approach advances carbon-aware inference by combining kernel-level graphs, hardware-aware features, and targeted data sampling to predict operational energy and CO2eq accurately.

Abstract

Throughout its lifecycle, a large language model (LLM) generates a substantially larger carbon footprint during inference than training. LLM inference requests vary in batch size, prompt length, and token generation number, while cloud providers employ different GPU types and quantities to meet diverse service-level objectives for accuracy and latency. It is crucial for both users and cloud providers to have a tool that quickly and accurately estimates the carbon impact of LLM inferences based on a combination of inference request and hardware configurations before execution. Estimating the carbon footprint of LLM inferences is more complex than training due to lower and highly variable model FLOPS utilization, rendering previous equation-based models inaccurate. Additionally, existing machine learning (ML) prediction methods either lack accuracy or demand extensive training data, as they inadequately handle the distinct prefill and decode phases, overlook hardware-specific features, and inefficiently sample uncommon inference configurations. We introduce \coo, a graph neural network (GNN)-based model that greatly improves the accuracy of LLM inference carbon footprint predictions compared to previous methods.
Paper Structure (13 sections, 52 equations, 13 figures, 7 tables, 1 algorithm)

This paper contains 13 sections, 52 equations, 13 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. LLMCO2
  5. Graph Embedding
  6. Focused Energy Data Sampling
  7. Evaluation
  8. Experimental methodology
  9. Operational energy results
  10. Ablation studies
  11. User Case Studies
  12. Conclusion
  13. The kernels in a transformer layer

Figures (13)

  • Figure 1: A decoder-only LLM's autoregressive inference consists of prefill and decode phases.
  • Figure 2: Comparing the energy of 2 phases in a Bloom-3B inference on an Nvidia L4 GPU.
  • Figure 3: An example of a all-reduce kernel running on four GPUs.
  • Figure 4: The kernels in a transformer layer.
  • Figure 5: The memory and network Roofline models for an Nvidia V100 GPU.
  • ...and 8 more figures