Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aging-aware CPU Core Management for Embodied Carbon Amortization in Cloud LLM Inference

Tharindu B. Hewage, Shashikant Ilager, Maria Rodriguez Read, Rajkumar Buyya

TL;DR

This work addresses the embodied carbon footprint, particularly CPU embodied emissions, in large-scale cloud LLM inference. It proposes aging-aware CPU core management that leverages underutilization patterns to halt aging on deep-idled cores and to even out aging across active cores, thereby extending CPU lifespan. Evaluations on production Azure traces and a Microsoft splitwise-sim based simulator demonstrate substantial reductions in yearly embodied carbon (up to 37.67% at p99) and in CPU underutilization (about 77%), with minimal impact on service quality. The approach is implemented in an open-source simulator and offers a practical path to decarbonize cloud LLM inference without sacrificing performance.

Abstract

Broad adoption of Large Language Models (LLM) demands rapid expansions of cloud LLM inference clusters, leading to accumulation of embodied carbon$-$the emissions from manufacturing and supplying IT assets$-$that mostly concentrate on inference server CPU. This paper delves into the challenges of sustainable growth of cloud LLM inference, emphasizing extended amortization of CPU embodied over an increased lifespan. Given the reliability risks of silicon aging, we propose an aging-aware CPU core management technique to delay CPU aging effects, allowing the cluster operator to safely increase CPU life. Our technique exploits CPU underutilization patterns that we uncover in cloud LLM inference by halting aging in unused cores and even-outing aging in active cores via selective deep idling and aging-aware inference task allocation. Through extensive simulations using real-world Azure inference traces and an extended LLM cluster simulator from Microsoft, we show superior performance of our technique over existing methods with an estimated 37.67\% reduction in yearly embodied carbon emissions through p99 performance of managing CPU aging effects, a 77\% reduction in CPU underutilization, and less than 10\% impact to the inference service quality.

Aging-aware CPU Core Management for Embodied Carbon Amortization in Cloud LLM Inference

TL;DR

This work addresses the embodied carbon footprint, particularly CPU embodied emissions, in large-scale cloud LLM inference. It proposes aging-aware CPU core management that leverages underutilization patterns to halt aging on deep-idled cores and to even out aging across active cores, thereby extending CPU lifespan. Evaluations on production Azure traces and a Microsoft splitwise-sim based simulator demonstrate substantial reductions in yearly embodied carbon (up to 37.67% at p99) and in CPU underutilization (about 77%), with minimal impact on service quality. The approach is implemented in an open-source simulator and offers a practical path to decarbonize cloud LLM inference without sacrificing performance.

Abstract

Broad adoption of Large Language Models (LLM) demands rapid expansions of cloud LLM inference clusters, leading to accumulation of embodied carbonthe emissions from manufacturing and supplying IT assetsthat mostly concentrate on inference server CPU. This paper delves into the challenges of sustainable growth of cloud LLM inference, emphasizing extended amortization of CPU embodied over an increased lifespan. Given the reliability risks of silicon aging, we propose an aging-aware CPU core management technique to delay CPU aging effects, allowing the cluster operator to safely increase CPU life. Our technique exploits CPU underutilization patterns that we uncover in cloud LLM inference by halting aging in unused cores and even-outing aging in active cores via selective deep idling and aging-aware inference task allocation. Through extensive simulations using real-world Azure inference traces and an extended LLM cluster simulator from Microsoft, we show superior performance of our technique over existing methods with an estimated 37.67\% reduction in yearly embodied carbon emissions through p99 performance of managing CPU aging effects, a 77\% reduction in CPU underutilization, and less than 10\% impact to the inference service quality.
Paper Structure (20 sections, 9 equations, 8 figures, 3 tables, 2 algorithms)

This paper contains 20 sections, 9 equations, 8 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background: Embodied Carbon Amortization in Cloud Servers through CPU Age Management
  4. Motivation: Impact of CPUs’ on Embodied Carbon in LLM Inference Clusters
  5. System Model and Problem Formulation
  6. System Model
  7. Aging Model
  8. Problem Formulation
  9. Extended Embodied Carbon Amortization through Aging-aware CPU Core Management in LLM Inference Clusters
  10. Task-to-Core Mapping
  11. Selective Core Idling
  12. Implementation
  13. Performance Evaluation
  14. Experiment Design and Setup
  15. Baselines:
  16. ...and 5 more sections

Figures (8)

  • Figure 1: Carbon footprint of A100x4 GPU server running per second inference application when powered by energy sources with different carbon intensity li24carbon-efficient-llm.
  • Figure 2: Distributions of running inference tasks in an LLM inference cluster of 22 H100 machines.
  • Figure 3: High-Level System Diagram of Aging-aware CPU Core Management in LLM Inference Clusters.
  • Figure 4: Changes in Operating temperature when 6 out of 12 cores set to deep idle in an Intel Xeon CPU. If awake, cores are 100% utilized.
  • Figure 5: Behavior of the piecewise Reaction Function ($F$) for utilization of the CPU.
  • ...and 3 more figures