Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems

Grant Wilkins, Srinivasan Keshav, Richard Mortier

TL;DR

This work tackles the high energy cost of LLM inference by introducing workload-based energy and runtime models for heterogeneous GPU-CPU systems. It combines extensive energy profiling across multiple open-source LLMs with regression-based models that capture how input and output token counts drive energy and latency. An offline routing optimization using these models demonstrates tunable trade-offs between energy, latency, and accuracy, with Mixtral SMoE models showing notable energy efficiency. The results enable energy-aware, per-workload scheduling in production environments, though the study is limited to a single HPC node and NVIDIA-centric measurements, suggesting avenues for online adaptations and broader hardware evaluation.

Abstract

The rapid adoption of large language models (LLMs) has led to significant advances in natural language processing and text generation. However, the energy consumed through LLM model inference remains a major challenge for sustainable AI deployment. To address this problem, we model the workload-dependent energy consumption and runtime of LLM inference tasks on heterogeneous GPU-CPU systems. By conducting an extensive characterization study of several state-of-the-art LLMs and analyzing their energy and runtime behavior across different magnitudes of input prompts and output text, we develop accurate (R^2>0.96) energy and runtime models for each LLM. We employ these models to explore an offline, energy-optimal LLM workload scheduling framework. Through a case study, we demonstrate the advantages of energy and accuracy aware scheduling compared to existing best practices.

Offline Energy-Optimal LLM Serving: Workload-Based Energy Models for LLM Inference on Heterogeneous Systems

TL;DR

This work tackles the high energy cost of LLM inference by introducing workload-based energy and runtime models for heterogeneous GPU-CPU systems. It combines extensive energy profiling across multiple open-source LLMs with regression-based models that capture how input and output token counts drive energy and latency. An offline routing optimization using these models demonstrates tunable trade-offs between energy, latency, and accuracy, with Mixtral SMoE models showing notable energy efficiency. The results enable energy-aware, per-workload scheduling in production environments, though the study is limited to a single HPC node and NVIDIA-centric measurements, suggesting avenues for online adaptations and broader hardware evaluation.

Abstract

The rapid adoption of large language models (LLMs) has led to significant advances in natural language processing and text generation. However, the energy consumed through LLM model inference remains a major challenge for sustainable AI deployment. To address this problem, we model the workload-dependent energy consumption and runtime of LLM inference tasks on heterogeneous GPU-CPU systems. By conducting an extensive characterization study of several state-of-the-art LLMs and analyzing their energy and runtime behavior across different magnitudes of input prompts and output text, we develop accurate (R^2>0.96) energy and runtime models for each LLM. We employ these models to explore an offline, energy-optimal LLM workload scheduling framework. Through a case study, we demonstrate the advantages of energy and accuracy aware scheduling compared to existing best practices.
Paper Structure (22 sections, 5 equations, 3 figures, 3 tables)

This paper contains 22 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Energy Consumption in AI Systems
  4. Energy-Aware Data Center Scheduling
  5. Methods
  6. LLM Selection
  7. Energy Profiling of Our Cluster
  8. NVIDIA GPUs
  9. AMD CPUs
  10. Problem Formulation
  11. LLM Inference Performance
  12. Experimental Strategy
  13. Vary Input Tokens
  14. Vary Output Tokens
  15. Randomization and Stopping Criteria
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Model performance against number of input tokens. Low variance renders error bars invisible.
  • Figure 2: Model performance against number of output tokens. Low variance renders error bars invisible.
  • Figure 3: Behavior under offline simulation as $\zeta$ varies. Round-robin and Random query assignment are indistinguishable.