Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications

Jiaxi Li, Yue Zhu, Eun Kyung Lee, Klara Nahrstedt

TL;DR

This study re-evaluates prefill–decode disaggregation for LLM serving by benchmarking multiple KV cache transfer paths and DVFS-based optimization against a new two-GPU colocated baseline. It systematically measures performance and energy across varying batch sizes and GPU frequencies, revealing that disaggregation benefits are conditional on load and transfer medium, and that stage-wise frequency scaling rarely reduces total energy. The results show disaggregated setups can incur higher energy consumption overall, though they may offer TPOT advantages for tight decoding requirements. The findings provide practical guidance on when disaggregation may be beneficial and which KV transfer strategies best balance performance and energy in real deployments.

Abstract

Different from traditional Large Language Model (LLM) serving that colocates the prefill and decode stages on the same GPU, disaggregated serving dedicates distinct GPUs to prefill and decode workload. Once the prefill GPU completes its task, the KV cache must be transferred to the decode GPU. While existing works have proposed various KV cache transfer paths across different memory and storage tiers, there remains a lack of systematic benchmarking that compares their performance and energy efficiency. Meanwhile, although optimization techniques such as KV cache reuse and frequency scaling have been utilized for disaggregated serving, their performance and energy implications have not been rigorously benchmarked. In this paper, we fill this research gap by re-evaluating prefill-decode disaggregation under different KV transfer mediums and optimization strategies. Specifically, we include a new colocated serving baseline and evaluate disaggregated setups under different KV cache transfer paths. Through GPU profiling using dynamic voltage and frequency scaling (DVFS), we identify and compare the performance-energy Pareto frontiers across all setups to evaluate the potential energy savings enabled by disaggregation. Our results show that performance benefits from prefill-decode disaggregation are not guaranteed and depend on the request load and KV transfer mediums. In addition, stage-wise independent frequency scaling enabled by disaggregation does not lead to energy saving due to inherently higher energy consumption of disaggregated serving.

Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications

TL;DR

This study re-evaluates prefill–decode disaggregation for LLM serving by benchmarking multiple KV cache transfer paths and DVFS-based optimization against a new two-GPU colocated baseline. It systematically measures performance and energy across varying batch sizes and GPU frequencies, revealing that disaggregation benefits are conditional on load and transfer medium, and that stage-wise frequency scaling rarely reduces total energy. The results show disaggregated setups can incur higher energy consumption overall, though they may offer TPOT advantages for tight decoding requirements. The findings provide practical guidance on when disaggregation may be beneficial and which KV transfer strategies best balance performance and energy in real deployments.

Abstract

Different from traditional Large Language Model (LLM) serving that colocates the prefill and decode stages on the same GPU, disaggregated serving dedicates distinct GPUs to prefill and decode workload. Once the prefill GPU completes its task, the KV cache must be transferred to the decode GPU. While existing works have proposed various KV cache transfer paths across different memory and storage tiers, there remains a lack of systematic benchmarking that compares their performance and energy efficiency. Meanwhile, although optimization techniques such as KV cache reuse and frequency scaling have been utilized for disaggregated serving, their performance and energy implications have not been rigorously benchmarked. In this paper, we fill this research gap by re-evaluating prefill-decode disaggregation under different KV transfer mediums and optimization strategies. Specifically, we include a new colocated serving baseline and evaluate disaggregated setups under different KV cache transfer paths. Through GPU profiling using dynamic voltage and frequency scaling (DVFS), we identify and compare the performance-energy Pareto frontiers across all setups to evaluate the potential energy savings enabled by disaggregation. Our results show that performance benefits from prefill-decode disaggregation are not guaranteed and depend on the request load and KV transfer mediums. In addition, stage-wise independent frequency scaling enabled by disaggregation does not lead to energy saving due to inherently higher energy consumption of disaggregated serving.
Paper Structure (22 sections, 5 figures)

This paper contains 22 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Inference
  4. Prefill and Decode Disaggregation
  5. KV Cache Sharing and Reusing
  6. Energy-efficient LLM Inference
  7. Related Work and Our Motivation
  8. Unexplored Baseline Configurations
  9. Benchmarking KV Cache Transfer Paths
  10. Benchmarking Disaggregated Frequency Scaling
  11. Motivation Summary
  12. Benchmark Methodology
  13. Overview
  14. Hardware
  15. Software Stack
  16. ...and 7 more sections

Figures (5)

  • Figure 1: TTFT and TPOT performance as batch size increases across different experimental setups.
  • Figure 2: Prefill and decode throughput as batch size increases across different experimental setups.
  • Figure 3: Total energy consumption as batch size increases across different experimental setups.
  • Figure 4: Energy contributed by each hardware component as batch size increases across different experimental setups.
  • Figure 5: TTFT-energy and TPOT-energy Pareto frontiers across different experimental setups. The inference workload includes a batch of 16 requests with input size 16,384 and output size 256.